I recently saw a post by James Gosling (via David Strommer ) called the Eight Fallacies of Distributed Computing . These are eight assumptions on network almost anyone new to distributed computing assumes  which proves to be wrong in the long run (and thus cause big problems and headaches).

I thought I'd try a to complement this list by adding few realities on distributed systems and data

1.  Expect a certain level of entropy in the system -  Sites are never fully synchronized (unless you stop new data from pouring in)
2. You can only afford to cache immutable data
3. It very very hard to be able to scale indefinitely
4. Observing global state is only possible via control messages
5. It is hard to achieve distributed consensus (membership in a cluster, total order, commitment etc.)
6. Expect to debug by log-files

There are probably many others - but these are the first few that came to mind :)