Right, but as I just said obeying a disallow directive isn't legally mandatory, so it doesn't mean much.

The difference depends on context, of course.

Generally speaking there are several cases to consider:

(1) Site requires agreeing on terms of service before browser can access content. In this case, scraping is a clear violation.

(2) Site terms of service forbid scraping content, but human visitors can view content and ...
(2a) site takes technical measures to exclude bots. In this case scraping is a no-no, but for a different reason: it violates the Computer Fraud and Abuse Act.
(2b) site takes no technical measures to exclude bots. In this case, the answer is unclear, and may depend on the specific jurisdiction (e.g. circuit court).

(3) Site has a robots.txt file and ...
(3a) robots.txt allows scraping. In this case, even if the terms of service forbid scraping, the permission given here helps the scraper's defense.
(3b) robots.txt forbids scraping. In this case obeying robots.txt isn't in itself legally mandatory, but it may affect your case if the site takes other anti-scraping measures.

Not *explicitly*. Offering such a database would be an invitation for people to look at the whole data broker industry. So what you, as a databroker who tracks and piegeonholes every human being who uses the Internet to a fare-the-well, do to tap into the market for lists of gullible yokels? You offer your customer, literally anyone with money, the ability to zero in on the gullible by choosing appropriate proxies.

For example, you can get a list of everyone who has searched for "purchasing real estate with no money down". Sad people who buy colloidal silver and herbal male enhancement products. People who buy terrible crypto assets like NFTs and memecoins. Nutters who spend a lot of time on conspiracy theory sites.

It's kind of like doxxing someone. You might not be able to find out directly that John Doe lives on Maple St and works for ACME services, but you can piece it together by the traces he leaves online. Only you do it to populations wholesale.

wait a week or two and the details will change completely.

Trump is nothing if not mercurial. His fans will tell you he's playing 11 dimensional chess... I have my doubts, but let's say that's true. The problem is that when it comes to the economy it's not chess. It's more like basketball, and the President is the point guard calling plays, except the play being called keeps changing before the players can execute the last call. It's a tough time to be running a business, you can't plan out more than a couple of weeks.

15 years ago I listened to a radio interview with Australian scientists from the Murdoch Children's Research Institute on this issue. They explained how to safely introduce these foods to babies. I took these steps with both my kids at 6 months. They are now 13 and 9 and have always been allergy free. The only food that babies should never have is honey because of the risk of botulism. After 12 months, honey is safe.

1. Put a very small amount of peanut butter on the outside of the baby's cheek. Wait to see if there is reaction.
2. Repeat next day.
3. Next day, put a very small amount of peanut butter on their tongue. Wait to see if there is a reaction.
4. Repeat next day.
5. At this stage you should be pretty confident that peanuts are safe.
6. Repeat with other foods, eg egg, strawberries.

These days the advice is not to put the food on the baby's skin and instead to put it in a diluted form on the inside of the baby's lip.

Right, the economist refer to this as "externality". Fossil fuels aren't cheap, if you factor in the costs that people using them transfer to third parties. Theoretically, if the true cost of using fossil fuels were factored into every pound of coal or gallon of gasoline consumed, then we would use *exactly the right amount* of fossil fuels. Probably not zero, but not as much as we do when we pretend pollution isn't a cost.

That's what they've done. Or rather they've bought the politicians who create the regulatory frameworks. But if people woke up and realized they've been frog-boiled into giving away their privacy, then that would be prohibitively expensive.

I essentially made the argument that if we want capitalism to work the way we were taught in civics class it is supposed to, companies must be forced by regulation not to undermine the basic assumptions that lead to efficient operation of the free market.

I am neither here nor there on a basic income. I think it depends on circumstances, which of course are changing as more and more labor -- including routine mental labor -- is being automated. We are eventually headed to a world of unprecedented productive capacity and yet very little need for labor, but we aren't there yet.

Anybody who is pushing AI services, particularly *free* AI services, is hoping to mine your data, use it to target you for marketing, and use the service to steer you towards opaque business relationships they will profit from and you will find it complicated and inconvenient to extricate yourself from.

The question is -- ideas that are bad for *who*? This may be a very bad idea for you and me, but it is a very good idea for Microsoft, especially as, like their online services, they will make money off of us and it will be very inconvenient for us to opt out.

In civics-lesson style capitalism, which I'm all in favor of, companies compete to provide things for us that we want and we, armed with information about their products, services and prices, either choose to give them our business or to give our business to a competitor.

Not to say that stuff doesn't *ever* happen, but it's really hard to make a buck as a business that way. So what sufficiently large or well-placed businesses do is earn money *other* ways, by entangling consumers in business relationships that are opaque and which they don't have control over, may not even be fully aware they're signing on to, and which are complicated and awkward to extricate themselves from. In other words a well placed company, like Microsoft or Google or Facebook, will constantly be looking at ways to make money outside the rigorous demands of free market economics.

It wouldn't be cost-effective in China either were it not for state support.

There is no doubt that global free trade in commodities, in the absence of any government support, would be the most economically efficient thing to have. But China -- probably correctly -- identifies dependency on foreign supply chains for critical materials as a *security* issue. So they have indirect and direct subsidies, as well as state owned enterprises that operate on thin or even negative profit margins.

Since China does this kind of support on a scale nobody else does, China produces more rare earths than any other country, even though it is not particularly well endowed with deposits. This solves China's security problem with the reliability of the supply, but creates a security problem for other countries.

China thinks like Japan did before WW2, like empire building European countries did in the 1800s. Control over resources is a national security weapon, both for defense and offense.

The behavioral model you have isn't supported by data. When you raise the standard of living and food security of population, the fertility rate goes down. When you have nothing, children are economic assets whose labor can support the family. It's not a great option, but some people live in conditions where there are no good options.

I don't know what what means?