Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.
I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.
I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
A big factor is that Trump actually got rid of the prioritization scheme in deportation courts that processes violent criminals (and similar) more quickly than people caught with a broken taillight.
He had the net effect of increasing the amount of time a dangerous immigrant would be inside the country while the system got clogged with all sorts of cases that the American public broadly agrees hardly matter.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s
There were also more deportations the year before Trump's first term than the year of his first term.
Consider the chart of border encounters. This one shows the past through 2023 [0] and in 2024 we had 2.1 million encounters at the southern border [1]. It appears that deportations are a function of border encounters, and border encounters increased substantially during Biden's administration.
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action.
Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did
Really? It was very widely reported. Just wait until you see how many people Obama deported
I agree it's good we have archives of all the sensible takes on deporting criminals with due process and not mass rounding up people based on skin color or last name at a very high expense to the taxpayers with poor results.
I think this is one of those times where two people agree on the evidence but not the resulting position.
It's very easy to fight straw men like all progressives are against all deportation and border enforcement.
"I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did..."
I think this is an essentially meaningless number until you also include the total number of illegals that were let in during the same period for each administration.
The more important number is how many people attempted crossing, which jumped significantly during Biden. This probably has something to do with their phenomenally cratered economies during COVID, multiple destabilized governments in LatAm, and severe climate migration.
That's a big part of why the Biden administration (on day one) proposed a bill that created a large economic development grant for LatAm while also modernizing border security.
No, 85% of apprehensions don't result in release into the country. That is a number that you'll have a hard time substantiating. Allegedly Mayorkas said it to the CBP union, according to the CBP union.
Below is an actual analysis of actual data from the right-leaning Cato Institute. The real number is somewhere in the 40-50% range and has been stable through most administrations (and, shockingly, Trump released more illegal immigrants into the country than Biden).
In addition to letting immigrants into the country at a higher rate, Trump also specifically slowed down deportation processing for violent immigrants, keeping them in the country, free (as is required by the Constitution's Equal Protection and Due Process clauses) for longer.
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today.
The biggest loss I see is Organizations - Department of Energy, 5473 to 3647.
I also see under Bureaus - Energy Programs, 4347 to 2521.
These are overlapping categories (-1826 on both).
There are others, but they seem smaller, a few hundred at most.
Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th.
Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets.
Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site!
Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
It's crazy that this is what's focused on, but an entire administration of authoritarian control, corruption, and a final pardoning of the President's criminal family is completely ignored.
Whether in bad taste, a mistake, or a result of autism, it really doesn't matter.
"they're not getting rid of gov workers en mass, don't be ridiculous"
This was always the plan.
"they're not going to blanket pardon January 6th rioters, don't be ridiculous"
Some people that basically walked into the capitol without any weapons or being violent were imprisoned and not even given as much as a trial for 4 years. Don't you think they've done enough time? Or are you completely politically motivated?
"they're not going to deregulate industries and set us back decades in term of ecological progress, don't be ridiculous"
Biden should have de-regulated immediately after coming into office (a temporary measure), but he basically did nothing and completely fucked our economy.
"they're not scrubbing open gov data, don't be ridiculous"
Is this any different than any other administration or are you jumping to conclusions because Trump bad?
Perhaps the marker indicating where you are is supposed to work it's way up the list? I don't think what you've said here - in this thread - has been unreasonable.
That comment also seems very reasonable to me and probably doesn't deserve the response it got.
The other one does make sense to me though (but would be better in a different context) and echoes my own personal frustration with the "don't be ridiculous" dismissal of concern for things going on in the face of a blatant pattern (which perhaps you don't agree is a pattern with your implication that the higher things in that list haven't been dismissed as ridiculous then immediately happened).
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public.
Also from their own site at data.gov
> The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.
The part " to inform decisions by the public and policymakers" in particular stands out.
Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
Open Data initiatives generally build upon the premise of immutability.
Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down.
You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives.
Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org).
If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
I found https://uc3.cdlib.org/data-mirror/ which was a mirror a few years ago, but it's been taken down due to a lack of ongoing funding.
It would be a great start to have a mirror of just the metadata, to at least see what is being removed.
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.
I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.
I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
A big factor is that Trump actually got rid of the prioritization scheme in deportation courts that processes violent criminals (and similar) more quickly than people caught with a broken taillight.
He had the net effect of increasing the amount of time a dangerous immigrant would be inside the country while the system got clogged with all sorts of cases that the American public broadly agrees hardly matter.
It's like a cartoon version of a shitty manager.
https://www.themarshallproject.org/2019/07/16/trump-tried-to...
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s
There were also more deportations the year before Trump's first term than the year of his first term.
Consider the chart of border encounters. This one shows the past through 2023 [0] and in 2024 we had 2.1 million encounters at the southern border [1]. It appears that deportations are a function of border encounters, and border encounters increased substantially during Biden's administration.
0. https://www.nytimes.com/interactive/2023/10/29/us/illegal-bo...
1. https://www.migrationpolicy.org/news/fy2024-us-border-encoun....
Seems to be working for me: https://www.ice.gov/spotlight/statistics
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action.
Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
and they weren't token tariffs.
they were mostly ineffective tariffs, in the "tried so hard and got so far but in the end it doesn't even matter" sense, but he wasn't all talk
If you do a thing and it doesn’t work and don’t even try to fix it… still all talk.
Trump killed the Trans Pacific Partnership.
(This was a surprise to me. I'm grateful that I'm not at risk of being extradited to rot in a chinese prison for reverse engineering some IoT POS.)
How would TPP have put you at risk for Chinese prison?
The TPP was meant to economically counterbalance China by working closely with their neighbors.
Without it their incentive was to cut deals to work closer economically with China…
Trump’s isolationist inclinations empowers America’s enemies.
> I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did
Really? It was very widely reported. Just wait until you see how many people Obama deported
> Biden’s administration deported far more people than Trump’s did
It seems like Trump’s biggest problem is, like himself, the people he put in charge are C-players, so getting anything complex done is hard.
Yes and: Infighting.
I once reported to a guy that set all us minions (product managers) against each other. It was awful.
It's upsetting to the progressives but Biden and Obama both deported far more than Trump did.
Thankfully we have archives and youtube, entirely searchable, where from 2008-2012 the democrats were strongly for sending people back home.
I agree it's good we have archives of all the sensible takes on deporting criminals with due process and not mass rounding up people based on skin color or last name at a very high expense to the taxpayers with poor results.
I think this is one of those times where two people agree on the evidence but not the resulting position.
It's very easy to fight straw men like all progressives are against all deportation and border enforcement.
> It's upsetting to the progressives but Biden and Obama both deported far more than Trump did.
Is it ? Outside of the extreme progressive ideologies , what about the progressive platform would be against deportations?
"I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did..."
I think this is an essentially meaningless number until you also include the total number of illegals that were let in during the same period for each administration.
The more important number is how many people attempted crossing, which jumped significantly during Biden. This probably has something to do with their phenomenally cratered economies during COVID, multiple destabilized governments in LatAm, and severe climate migration.
That's a big part of why the Biden administration (on day one) proposed a bill that created a large economic development grant for LatAm while also modernizing border security.
[flagged]
Oof, rumor mill strikes again.
No, 85% of apprehensions don't result in release into the country. That is a number that you'll have a hard time substantiating. Allegedly Mayorkas said it to the CBP union, according to the CBP union.
Below is an actual analysis of actual data from the right-leaning Cato Institute. The real number is somewhere in the 40-50% range and has been stable through most administrations (and, shockingly, Trump released more illegal immigrants into the country than Biden).
In addition to letting immigrants into the country at a higher rate, Trump also specifically slowed down deportation processing for violent immigrants, keeping them in the country, free (as is required by the Constitution's Equal Protection and Due Process clauses) for longer.
https://www.cato.org/blog/new-data-show-migrants-were-more-l...
https://www.themarshallproject.org/2019/07/16/trump-tried-to...
It seems like some groups already expected this from efforts during the previous Trump admin. This article mentions a few, and more recent preparations: https://www.scientificamerican.com/article/scientists-scramb...
Do we have any idea what datasets have gone missing? More details please from anyone who has them.
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today.
The biggest loss I see is Organizations - Department of Energy, 5473 to 3647. I also see under Bureaus - Energy Programs, 4347 to 2521. These are overlapping categories (-1826 on both).
There are others, but they seem smaller, a few hundred at most.
Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th.
Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
This seems like a super important observation.
If the number of dataset saw a massive jump (50%), then back down a week later, that seems more like the correction of an error.
Maybe its related to climate change and advantages of renewables?
I know DoE does "other things", but I don't expect these to be public anyway.
I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
Common Crawl's latest crawl was Jan 12th-25th, and the index is available.
[flagged]
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets.
Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site!
Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
This seems like some extreme jumping to conclusions, especially if no politically-notable or popular data sets appear to be touched.
For example, suppose the list of New York schools during 2020 [0] is removed. Is that -1 to the total indicative of something nefarious?
[0] https://catalog.data.gov/dataset/2020-doe-high-school-direct...
"it's not a nazi salute, don't be ridiculous"
"they're not building a camp in Guantanamo, don't be ridiculous"
"they're not getting rid of gov workers en mass, don't be ridiculous"
"they're not going to blanket pardon January 6th rioters, don't be ridiculous"
"they're not going to deport international students who protested against Israel, don't be ridiculous"
"they're not going to deregulate industries and set us back decades in term of ecological progress, don't be ridiculous"
"they're not scrubbing open gov data, don't be ridiculous" < you are here
Indeed, it would be a shame to jump to conclusions, we have 0 hints of what's happening
"it's not a nazi salute, don't be ridiculous"
It's crazy that this is what's focused on, but an entire administration of authoritarian control, corruption, and a final pardoning of the President's criminal family is completely ignored.
Whether in bad taste, a mistake, or a result of autism, it really doesn't matter.
"they're not getting rid of gov workers en mass, don't be ridiculous"
This was always the plan.
"they're not going to blanket pardon January 6th rioters, don't be ridiculous"
Some people that basically walked into the capitol without any weapons or being violent were imprisoned and not even given as much as a trial for 4 years. Don't you think they've done enough time? Or are you completely politically motivated?
"they're not going to deregulate industries and set us back decades in term of ecological progress, don't be ridiculous"
Biden should have de-regulated immediately after coming into office (a temporary measure), but he basically did nothing and completely fucked our economy.
"they're not scrubbing open gov data, don't be ridiculous"
Is this any different than any other administration or are you jumping to conclusions because Trump bad?
> < you are here
Your guesses about me are hilariously wrong.
Ex: https://news.ycombinator.com/item?id=42850668 , https://news.ycombinator.com/item?id=42850761, https://news.ycombinator.com/item?id=42816654
So trim those excessive confidence levels, and consider what you might change to improve your process.
Perhaps the marker indicating where you are is supposed to work it's way up the list? I don't think what you've said here - in this thread - has been unreasonable.
That comment also seems very reasonable to me and probably doesn't deserve the response it got.
The other one does make sense to me though (but would be better in a different context) and echoes my own personal frustration with the "don't be ridiculous" dismissal of concern for things going on in the face of a blatant pattern (which perhaps you don't agree is a pattern with your implication that the higher things in that list haven't been dismissed as ridiculous then immediately happened).
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public.
Also from their own site at data.gov
> The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.
The part " to inform decisions by the public and policymakers" in particular stands out.
Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
Open Data initiatives generally build upon the premise of immutability.
Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down.
You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives.
Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org).
If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
I once read that occasionally a new Pharoah would make a point of smashing all prior evidence of a prior ruler(s). Obelisks, murals, etc.
agreed, this is very likely just incompetence or bureaucracy at play, i doubt anyone in tr*mp's administration cares about the data being removed