A simple connection between Entropy and Information

This article follows a simple example laid out by Jaynes (1996).
Jaynes’ example is one that shows how one’s computation of the change in entropy in a physical / chemical process depends on the precise variables that one uses to label the macro state. If you use different variables (say you are insensitive to properties that someone else does have the ability to measure) you can invent situations where the entropy change might (to one observer) be contrived to violate the Second Law of Thermodynamics while the other observer sees no such violation.
Let’s consider a vessel whose volume is V, with a diaphragm that separates it into two parts – volumes V_1 and V_2 with N_1 and N_1 molecules of each type, respectively. The “1” side is filled with Argon gas of type A_1 which is indistinguishable from the type of Argon gas filling side “2”, which we call type A_2, at least for the observer named Babu. However, Alisha, with her access to superior technology is indeed able to perceive the difference between the two types of Argon. The container is in close thermal contact with a heat bath that maintains a constant temperature T. The equilibrium condition (same temperature throughout and equal pressure) implies that n_1/V_1 =n_2/V_2 =(n_1+n_2)/V.
Alisha, in addition to her ability to notice the difference between A_1 and A_2 also has sole access to a material called Whiffnium, which is permeable to A_1 but impervious to A_2. During the course of her research, she has also discovered Whaffnium, a new material which is permeable to A_2, but impervious to A_1. Let’s suppose that Alisha constructs two (infinitesimally thin) pistons that are initially placed very close to each other, one piston made of Whiffnium and the other of Whaffnium, as in the picture below.


Let’s suppose that just enough of A_1 permeates through the Whiffnium so that the partial pressure of A_1 is the same in the left side of the container as well as the intercalated region (between the Whiffnium and Whaffnium pistons). Similarly, let’s assume that just enough of A_2 permeates through the Whaffnium into the intercalated region (between the pistons) so that the partial pressure of A_2 is the same in the intercalated region as well as on the right side of the container. Now, due to the unbalanced pressure of A_2 impinging upon the Whiffnium piston, it is reversibly moved to the left and the entropy change in A_2 is
 = n_2 k_B ln(V/V_2)
Similarly, the Whaffnium piston is reversibly moved to the right and the entropy change in A_1 is
= n_1 k_B ln(V/V_1)
The total entropy change is hence
n_1 k_B ln(V/V_1 )+n_2 k_B ln(V/V_2 )
All this is pretty logical from Alisha’s point of view, since she does see the two parts of the container as having different materials, A_1 and A_2. She understands that the entropy change in the container is a consequence of the heat flowing into the system from the heat bath.
However, Babu sees a conundrum. He sees the argon as one undifferentiated gas and so the initial and final states of the system are identical. However, the system has absorbed an amount of heat and converted all of it into work, in violation of the Second Law of Thermodynamics. In addition, he sees the entropy change as 0. This is, however, simply a reflection of the fact that the entropy is a function of the macrostate variables that one uses and if Babu has an insufficiently specified macrostate, then Alisha is simply able to manipulate phenomena to cause Babu to think he has observed a violation of the Second Law.
How much information would Babu need to in order to deduce the correct entropy change? In the initial state, if he knew about the sub-identities A_1 and A_2, the macrostate where A_1 is on the left and A_2 is on the right (side of the container) has the following number of (equally probable) microstates
=(V_1/v)^(n_1 ) (V_2/v)^(n_2 )
Where we have assumed that each molecule can be localized to a minimum volume v and we can do this for all the n_1 molecules in V_1 and the n_2 molecules in V_2.
In the final state, all the n_1+n_2 molecules are strewn about in the total volume V and the total number of microstates is
= (V/v)^(n_1+n_2)
So to specify the microstate, he needs to be communicated extra information (along the lines of traditional information theory)

I= \log_2((\frac{V}{v})^{n_1+n_2} ) - \log_2 ( (\frac{V_1}{v})^{n_1} (\frac{V_2}{v})^{n_2} )

= n_1 \log_2(V/V_1)+n_2 \log_2(V/V_2 )
Which is exactly (up to a multiplicative factor of k_B and difference between \ln and \log, just the same as the entropy change to separate the molecules into the different varieties.
Note that if Babu didn’t actually meet Alisha and had no idea that there were two varieties of Argon, his calculation for the number of microstates before and after would be identical, equal to (V/v)^(n_1+n_2 ) – this is because he doesn’t even think the diaphragm separating the two sides of the container is even necessary – they are the same materials and are in thermodynamic equilibrium with each other.
However, once Babu has this information, in the form of a detailed message, he will have been supplied with enough information to deduce completely (as far as Alisha’s abilities admit) to the situation with the two varieties, where he had zero before. Ergo, the extra information he needs is the entropy difference.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s