Principle of Maximum Entropy

INFORMATION-THEORETIC MEASURE AND PRINCIPLE OF MAXIMUM ENTROPY

Let P=(p₁ , p₂ ,…,p_n), be a probability distribution, then the first important measure of information was given by a pioneer Communication Engineer C.E. Shannon [2] in 1948. The amount of information is that amount by which the uncertainty in a situation characterized by a probability distribution ‘P’ is reduced, when it is known that which outcome will occur. The problem can be reduced in that of finding uncertainty associated with a probability distribution ‘P’. On the basis of some plausible postulates the measure of uncertainty of ‘P’ is deduced as

H (P) = - Sum(pi * ln(pi)) (1)

H(P) is known as a measure of entropy. The word entropy stands for ‘uncertainty’. H(P) satisfies most of the useful properties as, non-negativity, concavity, additivity, increasing with number of outcomes, maximum for uniform distribution, minimum for degenerate distribution and minimum value is zero etc. required to be satisfied by a measure of entropy. Here information and uncertainty are intrinsically related as

Information gained == uncertainty removed

Later E.T. Jaynes [3] used this measure by enunciating his Principle of Maximum Entropy (PME), according to which, out of all the probability distributions given, one should choose that probability distribution which maximizes H (P) and satisfy all the given constraints. In the absence of any constraints on p_i’s except natural constraints, the Maximum Entropy Probability Distribution (MEPD) is a uniform probability distribution. Further if additional information in terms of simple statistical moments of a random variable is prescribed, then we will get a probability distribution, which is ‘closest’ to a uniform probability distribution. In this sense, principle of maximum entropy is a powerful tool to analyse a situation, when partial information in terms of simple statistical moments is prescribed.

My QUESTION : HOW CAN THIS POWERFUL PRINCIPLE BE USED FOR PRODUCT DESIGN AND SYSTEM ANALYSIS?