Comment Should have read "or less" (Score 1) 144
100m data points *or less*
100m data points *or less*
here are my answers. Spreadsheets are used in several cases:
1) When you have a small-to-medium-sized dataset (100m data points) and want to do a particular set of calculations or draw a particular set of conclusions from it just once or twice—so that the time invested in writing code in R or something similar is less than the time needed just to bung a few formulas into a spreadsheet and get your results. Once you get into analyses or processes that will be repeated many times, it makes more sense to write code.
2) Similar case, when you need to work with essentially tabular database data, but the operations you're performing (basic filtering, extracting records based on one or two criteria, just handing data from one person to the next) are either so simple or will be repeated so rarely that a MySQL database is overkill and just emailing a file back and forth is easier.
3) When you are working with data as a part of a team, and certain members of the team that are specialists in some areas related to the data, or (for example) members of the team that are doing your data collections, aren't particularly computationally expert. Spreadsheets are hard for laymen, but it's doable—a dozen or two hours of training and people can get a general, flexible grasp of spreadsheets and formulae. It takes a lot longer for someone to become basically proficient with R, MATLAB, MySQL, Python, etc., and you really want those specialists to just be able to do what they do to or with the data, rather than focusing their time and effort on learning computational tools. Spreadsheets are general purpose and have a relatively shallow learning curve relative to lots of other technologies, but they enable fairly sophisticated computation to take place—if inefficiently at times. They're like a lowest-common-denominator of data science.
We use Spreadsheets all the time in what we do, mostly as a transient form. The "heavy hitting" and "production" data takes place largely in MySQL and R, but there are constant temporary/intermediate moments in which data is dumped out as a CSV, touches a bunch of hands that are really not MySQL or R capable, and then is returned in updated form to where in normally lives.
Regarding your number 2... Frequently get tampered with in transit? Really? I have, literally, never seen this....
You're lucky there. I see such tampering several times per day, and fixing the problem often takes a lot of time (and soto-voce swearing
The reason is that I deal with a lot of data that's "plain text", but is computer data of some sort, not a natural language like English (which is sorts stretching the meaning of "natural", but you know what I mean). Or it's in a human language, but not English, and the character encoding uses some 2-byte or longer characters.
The simplest example is computer source code. The tampering is often caused by the "punch-card mentality" coded into a lot of email software, which often doesn't allow lines longer than 80 (or 72) characters, and inserts line feeds to make everything fit. Many programming languages consider line feeds to mean something different than a space, usually "end of statement". Inserting a line feed in the middle of a statement thus changes the meaning, and very often introduces a syntax error.
Even nastier is the munging a lot of other plain-text data representation that mixes letters and numbers. Inserting spaces or a line feed in the middle of a token like "G2EF" usually destroys the meaning in a way that can't be corrected automatically at the receiving end. Usually the way to handle such tampering is to reply to the sender, saying "Can you send me that in quoted-printable or base-64 form?" And you try to teach everyone in the group that such data should always be encoded in a form that's immune to the idiocies of "smart" email handlers.
Text in UTF-8 form, especially Chinese and Japanese text, is especially prone to this sort of tampering, which often leaves the text garbled beyond recovery.
Anyway, there are lots of excuses for such tampering with email in ways that destroy the content. It's not always for nefarious reasons; it's just because the programmers only tested their email-handling code on English-language text. And because they're idiots who think that lines of text should never be longer than 80 (or 72) characters.
When I noticed that the address was the address of my machine, I did a quick find(1), but couldn't find the IMDB files or the takedown letter. Do you think I should contact Universal Pictures and ask them to send me another copy of the letter, so I can figure out which file to take down?
Actually, I noticed that all of our home machines (we have several, including tablets and smart phones) seem to have the same address. I guess that's to be expected, since ISPs only give us a single address, so we all have to use that silly NAT protocol and try to make sense of the confusion that it always creates. Anyway, I did look around on all of them, and still couldn't find anything with "Universal Pictures" inside. I did find a few files that contain "IMDB", but they're in the browsers' cache directories, and I got rid of those by simply telling the browsers to clear their cache(s).
But somehow I don't think this has taken care of the problem. So who should I contact at Universal Pictures to make sure we get a copy of the letter and purge our machines of their files?
(And for the benefit of many
If you click through to the "article," you'll get a long list of ten-word sentences formatted as tweets. When did this become an even remotely acceptable way of presenting something?
Good lord. If you work for Google, can't you figure out how to create a blog? My mother did it.
What rate of return would convince you to put your money in an investment if you knew it was going to be 10 years before you received the first dollar back - and there was a 90%+ chance of failure to boot?
Funny thing; those numbers were used back in the 1980s, with interesting results. The topic wasn't drugs, though, but rather solid-state manufacturing, and very similar numbers were widely quoted in east Asia. At the time, it was generally estimated that to build a new solid-state facility would require several billion dollars, and would take around a decade to become profitable, due to the extreme difficulty of achieving the required low level of contaminants inside the equipment. Much of the decade would be spent making test runs, discovering that the output was useless because of some trace contaminant in one part of the process, and redesigning the setup to get past yet another failure. Success wasn't predictable; the 10-year estimate was just the minimum.
But people in east Asia (mostly Japan and Korea) argued publicly that the American companies that controlled most of the production at the time wouldn't be able to get funding for new factories, because American investors would refuse to invest so much money in something with no payoff for a decade. If Asian investors would step in and support the effort, in 10 years they could own the world's solid-state industry. Enough people with money (including government agencies) listened, made the gamble, and a decade longer, they owned the industry.
It's probably just a matter of time before the American drug industry goes the same way. Would you invest in something with no payoff for a decade or more, and wasn't even guaranteed to pay off then because nobody had yet created the drugs that might be created? If you guess that few US (or EU) investors will do this, you're likely right.
In particular, the Republican US Congress is highly likely to continue its defunding of academic basic research, partly due to mistrust of investments that won't pay off during their current terms in office, and also due to a serious religion-based dislike of the biological sciences in general. Without the basic research, the only "new" drugs patented by industry will continue to be mostly small tweaks of existing drugs, which under US law qualify as new, patentable products.
Of course, this is all a bunch of tenuous guesses, based on past behavior of the players. That's what investment is usually like. It's entirely possible that they'll wise up, and not abandon the drug industry the way they abandoned the electronics industry. The US does actually have a few solid-state production facilities, after all, though they're now a small part of the market.
But, as the above poster said, would you be willing to gamble your investment money on the hope that US private drug makers will support the research that the US government is getting out of? Remember that, to corporate management, scientific research appears to have a record of 90% failure; i.e., 90% of funded research projects fail to produce a patentable and marketable product. This is the nature of research, which only discovers facts and theories, not products, and where the outcome of a study is unpredictable before the fact. (If it were predictable, it wouldn't be called "research", it'd be "development".
Why does a car have a wireless system, and why is this wireless system accessible from outside the car?
So that the manufacturer can access the car, collect data on where and how it's been driven, and sell that information to anyone willing to pay for it.
The idea of sending "data" to the car was an afterthought, when they realized it could be useful for things like disabling a car that's behind on the payments.
Note that both of these motives contain the string "pay". That's the hint you need to figure out the other intended uses.
Because the number one thing openness generates is chaos and multiple competing claims about reality. Say, many Linux distributions, each claiming to be great, and in fact, many variants of Linux distributions often with many versions and many wrinkles, and many varations of packages, libraries, and so on.
If you want to build or customize things, openness is great. If you just one to pick something up, use it, and move on, a huge amount of confusion, overhead, and pain is involved in trying to pick the "right" version (particularly if you're unfamiliar with openness and wrongheadedly looking for the "real" version, as many early Linux dabblers were) and get it to work quickly and easily.
There is thus a huge amount of value added by anyone that quells the chaos—even in a tiny sphere or product—and that can quickly, clearly, and succinctly explain to users just what their version does, without ambiguity either within itself as an instance or over time. The nature of the beast—this value is the result of "closing the openness," if you will, means that it can't be opened, or the value will be lost.
End users want operating systems and devices that are not open systems with unclear edges that bleed into the ecosystem, but rather a single, coherent, object or product that they can acquire, use in predictable and stable ways, and then lay down once again. They want systems and devices about which books can be written (and bought, and referred to months down the road) without quickly becoming obsolete, and with the minimal risk that this book or that add-on that they purchase will fail to work becuase they'd misconstrued the incredibly subtle differences and variations in product naming, versioning, and so on.
In short, massive openness is incredibly generative and creative, but leaves in place a systems/software/hardware version of the "last mile problem" for computing. Having a fabulous network is one thing, but consumers just want one wire coming into the house, they want it to work, they want it to be predictable and compatible with what they have, and they want to know just where it is and what its properties, limits, and costs are. They are not interested in becoming engineers, the technology they use is only useful to them as a single, tiny, and managable facet of the larger ecosystem that is their life.
This "last mile problem" cannot be solved with openness in hardware or software any more than the last mile problem for wired providers can be solved by opening up all of urban geography to any comers and saying "lay all the cable you want, anywhere you want, to and from any building or system!" First off, it would result in a mess of wires (not un-analagous to what we see across much of free software's development space) and next because most consumers wouldn't be able to make heads or tails of it, much less make a choice, and they'd probably resent the complexity in their backyard and try to do away with it.
Openness leads to closedness because to the extent that openness dominates in the development and engineering space, closedness increases as critical need for carrying whatever is developed to the average consumer space, in precisely the same measure.
Automotive control interfaces change all of the time.
Really? The "control interface" of my '81 Ford is the same as the day it was purchased.
Well, the auto makers have "fixed" that problem in their latest models. They now have those little "onboard computers" that constantly scan many of the controls and figure out how to map them to physical actions. This means that any "upgrade" to the software can change the functioning of all the controls. You can think you're just getting an upgrade to improve the mileage, but that upgrade can flip the meaning of the turn-signal controls.
Some of the latest models have wifi, so they can do upgrades while you're traveling. We'll probably soon be hearing of accidents caused by a sudden change in meaning of what the driver did with the controls. (Yes, they may say the upgrades won't happen while the car is moving. What that means is that if you stop at a stop sign or light, when you start moving again, the controls may have silently changed. And if you think they wouldn't do upgrades without your permission, you haven't been paying attention.)
If computer-industry history is any guide, it'll probably take decades for all this to settle down to an intuitive, reliable auto UI. And the security problems still won't be solved, so your car can be taken over at any moment by "hackers" - or the police - or your insurance company.
(I wish I were joking
If you're going to link to a site, you should link to one that involves actual scientists using actual science.
Nah; if we do that, they'll all just agree with each other. (Haven't you been paying attention.
That would really baffle anyone in the 95% of the human population who's not a literate, native speaker of English.
Curious how you decided English has such a poor showing across the world?
English doesn't have to be your primary language to be fluent in it.
Well,yeah, but that's balanced out by the large population of native English speakers with a poor command of the language.
We're seeing a bit of that here on
To do nothing is to be nothing.
