Optimization with EM and Expectation-Conjugate-Gradient by dfsiopmhy6

									      Optimization with EM and Expectation-Conjugate-Gradient

Ruslan Salakhutdinov                                                              rsalakhu@cs.toronto.edu
Sam Roweis                                                                          roweis@cs.toronto.edu
Department of Computer Science, University of Toronto
6 King’s College Rd, M5S 3G4, Canada
Zoubin Ghahramani                                                                   zoubin@gatsby.ucl.ac.uk
Gatsby Computational Neuroscience Unit, University College London
17 Queen Square, London WC1N 3AR, UK

                     Abstract                             In spite of tremendous success of the EM algorithm in
                                                          practice due to its simplicity and fast initial progress,
    We show a close relationship between the
                                                          some authors [8] have argued that the speed of EM
    Expectation - Maximization (EM) algorithm
                                                          convergence can be extremely slow, and that more
    and direct optimization algorithms such as
                                                          complicated second-order methods should generally be
    gradient-based methods for parameter learn-
                                                          favored to EM. Many methods have been proposed
    ing. We identify analytic conditions under
                                                          to enhance the convergence speed of the EM algo-
    which EM exhibits Quasi-Newton behavior,
                                                          rithm, mostly based on conventional optimization the-
    and conditions under which it possesses poor,
                                                          ory [6, 7]. Several authors [8, 1] have also proposed hy-
    ﬁrst-order convergence. Based on this analy-
                                                          brid approaches for ML learning, advocating switching
    sis, we propose two novel algorithms for max-
                                                          to a Newton or Quasi-Newton method after perform-
    imum likelihood estimation of latent variable
                                                          ing several EM iterations. All of these approaches, al-
    models, and report empirical results show-
                                                          though sometimes successful in terms of convergence,
    ing that, as predicted by the theory, the pro-
                                                          are much more complex than EM, and diﬃcult to an-
    posed new algorithms can substantially out-
                                                          alyze; thus they have not been popular in practice.
    perform standard EM in terms of speed of
    convergence in certain cases.                         Our goal in this paper is to contrast the EM algorithm
                                                          with a direct gradient-based optimization approach.
                                                          As a concrete alternative, we present an Expectation-
                                                          Conjugate-Gradient (ECG) algorithm for maximum
1. Introduction                                           likelihood estimation in latent variable models, and
The problem of Maximum Likelihood (ML) parameter
                                                          show that it can outperform EM in terms of conver-
estimation for latent variable models is an important
                                                          gence in certain cases. However, in other cases the
problem in the area of machine learning and pattern
                                                          performance of EM is superior. To understand these
recognition. ML learning with unobserved quantities
                                                          behaviours, we study the convergence properties of the
arises in many probabilistic models such as density
                                                          EM algorithm and identify analytic conditions under
estimation, dimensionality reduction, or classiﬁcation,
                                                          which EM algorithm exhibits Quasi-Newton conver-
and generally reduces to a relatively hard optimization
                                                          gence behavior, and conditions under which it pos-
problem in terms of the model parameters after the
                                                          sesses extremely poor, ﬁrst-order convergence. Based
hidden quantities have been integrated out.
                                                          on this analysis, we introduce a simple hybrid EM-
A common technique for ML estimation of model             ECG algorithm that switches between EM and ECG
parameters in the presence of latent variables is         based on estimated quantities suggested by our anal-
Expectation-Maximization (EM) algorithm [3]. The          ysis. We report empirical results on synthetic as well
EM algorithm alternates between estimating the un-        as real-world data sets, showing that, as predicted by
observed variables given the current model and reﬁt-      the theory, this simple algorithm almost never per-
ting the model given the estimated, complete data. As     forms worse than standard EM and can substantially
such it takes discrete steps in parameter space similar   outperform EM’s convergence.
to to ﬁrst order method operating on the gradient of
a locally reshaped likelihood function.
2. Linear and Newton Convergence of                                                 M-step has a single unique solution.2
   Expectation Maximization                                                         The important consequence of the above analysis is
We ﬁrst focus on the analysis of the convergence                                    that (when the expected complete log-likelihood func-
properties of the Expectation-Maximization (EM)                                     tion has a unique optimum), EM has the appealing
algorithm. Consider a probabilistic model of ob-                                    quality of always taking a step Θ(t+1) −Θt having pos-
served data x which uses latent variables y. The                                    itive projection onto the true gradient of the objective
log-likelihood (objective function) can be written as                               function L(Θt ). This makes EM similar to a ﬁrst order
a diﬀerence between expected complete log-likelihood                                method operating on the gradient of a locally reshaped
and negative entropy terms:                                                         likelihood function.
L(Θ) = ln p(x|Θ) =                p(y|x, Ψ) ln p(x|Θ)dy =
                                                                                For maximum likelihood learning of a mixture of Gaus-
  p(y|x, Ψ) ln p(x, y|Θ)dy −                 p(y|x, Ψ) ln p(y|x, Θ)dy               sians model using the EM-algorithm, this positive def-
  = Q(Θ, Ψ) − H(Θ, Ψ)                                                               inite transformation matrix P (Θt ) was ﬁrst described
                                                                                by Xu and Jordan[15]. We extended their results by
 The EM algorithm implicitly deﬁnes a mapping:                                      deriving the explicit form for the transformation ma-
M : Θ → Θ from parameter space to itself, such that                                 trix for several other latent variables models such as
Θt+1 = M (Θt ). If iterates of Θt converge to Θ∗ (and                               Factor Analysis (FA), Probabilistic Principal Compo-
M (Θ) is continuous), then Θ∗ = M (Θ∗ ), and in the                                 nent Analysis (PPCA), mixture of PPCAs, mixture of
neighbourhood of Θ∗ , by a Taylor series expansion:                                 FAs, and Hidden Markov Models [10].3
             Θt+1 − Θ∗ = M (Θ∗ )(Θt − Θ∗ )                                    (1)   We can further study the structure of the transfor-
                                                                                mation matrix P (Θt ) and relate it to the convergence
where M (Θ∗ ) = ∂M |Θ=Θ∗ . Therefore EM is essen-
                     ∂Θ                                                             rate matrix M . Taking the negative derivatives of
tially a linear iteration algorithm with a convergence                              both sides of (2) with respect to Θt , we have:
rate matrix M (Θ∗ ) (which is typically nonzero).
                                                                                I − M (Θt ) = −P (Θt )      L (Θ
                                                                                ) − P (Θt )S(Θt )   (4)
For most objective functions, the EM step Θ(t+1) −Θ(t)
                                                                                ∂ 2 L(Θ)
in parameter space and true gradient can be related by                              where S(Θt ) =        ∂Θ2 |Θ=Θ is the Hessian
                                                                                t                         of the
a transformation matrix P (Θt ), that changes at each                                                            t    ∂Θt+1
                                                                                objective   function, Mij (Θ ) = ∂Θt is the
iteration:                                                                                j
                                                                                output derivative matrix for the EM mapping and
               Θ(t+1) − Θ(t) = P (Θt )              L (Θ
                                                               )              (2)   P (Θt ) = ∂P (Θ) |Θ=Θt is the tensor derivative of P (Θt )
                                                                                with respect to Θt . In ”ﬂat” regions of L(Θ), where
(We deﬁne L (Θt ) = ∂L(Θ) |Θ=Θt .) Under certain                                                                         t
                           ∂Θ                                                         L (Θ) approaches zero (and P (Θ ) does not become
conditions, the transformation matrix P (Θt ) is                                    inﬁnite), the ﬁrst term on the RHS of equation (4)
guaranteed to be positive deﬁnite with respect to the                               becomes much smaller than the second term, and the
gradient.1 In particular, if                                                        transformation matrix becomes a rescaled version of
 C1: Expected complete log-likelihood Q(Θ, Θt ) is well-                            the negative inverse Hessian:
     deﬁned, and diﬀerentiable everywhere in Θ. and
                                                                                P (Θt ) ≈ I − M (Θt )           − S(Θt )            (5)
 C2: For any ﬁxed Θt = Θ(t+1) , Q(Θ, Θt ) has only a
         single critical point along any direction in Θ, located
                                                                                In particular, if the EM algorithm iterates converge to
         at the maximum Θ = Θt+1 ; then
                                                                                a local optima at Θ∗ , then near this point (i.e. for suﬃ-
                L (Θ       )P (Θt )   L (Θ
                                                 ) > 0 ∀Θt                    (3)   ciently large t) EM may exhibit Quasi-Newton conver-
                                                                                gence behavior. This is also true in “plateau” regions
The second condition C2 may seem very strong. How-                                  where the gradient is very small even if they are not
ever, for the EM algorithm C2 is satisﬁed whenever the                              near a local optimum.
       Note that    Q (Θ
                                 )(Θ(t+1) − Θt ), where            Q (Θ
                                                                              ) =   The nature of the Quasi-Newton behavior is controlled
∂Q(Θ,Θ )
         |Θ=Θt is the directional derivative of function                            by the eigenvalues of the matrix M (Θt ). If all eigen-
Q(Θ, Θ ) in the direction of Θ(t+1) − Θt .
                                                    C1 and                          values tend to zero, then EM becomes a true Newton
C2 together imply that this quantity is positive, oth-                                 2
                                                                                In particular C2 holds for any exponential family
erwise by the Mean Value Theorem (C1) Q(Θ, Θt )
                                                                                model, due to the well-known convexity property of
would have a critical point along some direction, lo-
                                                                                Q(Θ, Θt ) for these models.
cated at a point other than Θt+1 (C2).               By us-                            3
                         t       ∂Q(Θ,Θt )
                                                                                We also derived the general form of the transformation
ing the identity     L (Θ ) =        ∂Θ
                                           |Θ=Θt , we have                          matrix for the exponential family models in term of their
      t     t      t           t    (t+1)
  L (Θ )P (Θ ) L (Θ ) =    Q (Θ )(Θ       − Θt ) > 0.                               natural parameters.
                                                  10                                                                   6
                                                                                EM                     5
            µ1                    µ2                                                                                3

                                                  −5                                                                   1

                                                                                7.7                                 0
 −9    −6    −3       0       3        6   9       −10                 −5                0        5               10         −6   −5   −4     −3   −2     −1       0
                                                  10                                                                       1.5
                                                                                GRADIENT         9.0
                                                                             EM                                             1

                                                   0                                                                        0
                  µ       µ
                  1       2
                                                  −5                                                        8.5

                                                                       9.1         8.0
                                                  −10                                                                  −1.5
  −6        −3        0           3        6       −10                 −5                0        5               10    −1.5      −1   −0.5    0    0.5        1       1.5

Figure 1. Contour plots of the likelihood function L(Θ) for MoG examples using well-separated (upper panels) and not-
well-separated (lower panels) one-dimensional data sets. Axes correspond to the two means. The dashdot line shows the
direction of the true gradient L (Θ), the solid line shows the direction of P (Θ) L (Θ) and the dashed line shows the
direction of (−S)−1 L (Θ). Right panels are blowups of dashed regions on the left. The numbers indicate the log of the
l2 norm of the gradient. Note that for the ”well-separated” case, in the vicinity of the maximum, vectors P (Θ) L (Θ)
and (−S)−1 L (Θ) become identical.
method, rescaling the gradient by exactly the nega-                                                           case of ﬁtting a mixture of Gaussians model to well-
tive inverse Hessian. As the eigenvalues tend to unity,                                                       clustered data – for which EM exhibits Quasi-Newton
EM takes smaller and smaller stepsizes, giving poor,                                                          convergence – and not-well-clustered data, for which
ﬁrst-order, convergence.                                                                                EM is slow. As we will see from the empirical results
                                                                                of the later sections, many other models also show this
Dempster, Laird, and Rubin [3] showed that if EM
                                                                                same eﬀect. For example, when Hidden Markov Mod-
iterates converge to Θ∗ , then
                                                                                els or Aggregate Markov Models [11] are trained on
∂M (Θ)                                 ∂ 2 H(Θ,Θ∗ )                  ∂ 2 Q(Θ,Θ∗ )                             very structured sequences, EM exhibits Quasi-Newton
 ∂Θ |Θ=Θ                  =                         |Θ=Θ∗                         |Θ=Θ∗
                                            ∂Θ2                           ∂Θ2
                                                                                behavior, in particular when the state transition ma-
which can be interpreted as the ratio of missing                                                              trix is sparse and the output distributions are almost
information to the complete information near the                                                              deterministic at each state.
local optimum. Thus, in the neighbourhood of a                                                                The above analysis and experiments motivates the use
solution (for suﬃciently large t):                                                                            of alternative optimization techniques in the regime
                                                         −1                                     −1
                                       ∂2H      ∂2Q                                                           where missing information is high and EM is likely
P (Θt ) ≈ I −                                                   |Θ=Θt             − S(Θt )
                                       ∂Θ2      ∂Θ2                                                           to perform poorly. In the following section, we an-
                                                    (6)                                                       alyze exactly such an alternative, the Expectation-
This formulation of the EM algorithm has a very in-                                                           Conjugate Gradient (ECG) algorithm, a simple direct
teresting interpretation which is applicable to any la-                                                       optimization method for learning the parameters of
tent variable model: When the missing information is                                                          latent variables models.
small compared to the complete information, EM ex-
hibits Quasi-Newton behavior and enjoys fast, typically                                                       3. Expectation-Conjugate-Gradient
superlinear convergence in the neighborhood of Θ∗ . If
                                                                                (ECG) Algorithm
fraction of missing information approaches unity, the
eigenvalues of the ﬁrst term (6) approach zero and EM                                                         The key idea of the ECG algorithm is to note
will exhibit extremely slow convergence.                                                                      that if we can easily compute the derivative
                                                                                ∂Θ ln p(x, z|Θ) of the complete log likelihood,
Figure 1 illustrates the above results in the simple
then knowing the posterior p(z|x, Θ) we can com-                                                                                oped to escape from such conﬁgurations. It turns out
pute the exact gradient        L (Θ).   In particular:                                                                                that direct optimization methods such as ECG may
  L (Θ) = z p(z|x, Θ) ∂Θ log p(x, z|Θ)dz. This exact                                                                                also avoid this problem because of the nonlocal nature
gradient can then be utilized in any standard manner,                                                                                of the line search. In many of our experiments, ECG
for example to do gradient (as)descent or to control                                                                                actually converges to a better local optimum than EM;
a line search technique. (Note that if one can derive                                                                                ﬁgure 2 illustrates exactly such case.
exact EM for a latent variable model, then one can
always derive ECG by computing the above integral                                                                                4. Hybrid EM-ECG Algorithm
over hidden variables.) As an example, we describe a
conjugate gradient algorithm:                                                                                As we have seen, the relative performance of EM ver-
                                                                                sus direct optimization depends on the missing infor-
Expectation-Conjugate-Gradient algorithm:                                                                                mation ratio for the given model and data set. The key
Apply a conjugate gradient optimizer to L(Θ), perform-                                                                                to practical speedups is the ability to design a hybrid
ing an “EG” step whenever the value or gradient of                                                                                algorithm that can estimate the local missing informa-
L(Θ) is requested (e.g. during a line search).                                                                                tion ratio M (Θt ) to detect whether to use EM or a
The gradient computation is given by                                                                                direct approach such as ECG. Some authors have at-
         • E-Step: Compute posterior p( | , Θt ) and log-

                                                                                tacked this problem by ﬁnding the top eigenvector of
           likelihood L(Θ) as normal.                                                                                ∂M (Θ)             t               ∗
                                                                                ∂Θ |Θ=Θ as Θ approaches Θ using conventional
         • G-Step:                      L (Θ        )=       p( | , Θt ) ∂Θ log p( , |Θ)d

                                                                                ¡                                         ¡

                                                                                numerical methods, such as ﬁnite-diﬀerence approxi-
                                                                                mations, or power methods [4]. These approaches are
When certain parameters must obey positivity or lin-                                                                                computationally intensive and diﬃcult to implement,
ear constraints, we can either modify our optimizer                                                                                and thus they have not been popular in practice.
to respect the constraints, or we can reparameterize
to allow unconstrained optimization. In our exper-                                                                                Here, we propose using the entropy of the posterior
iments, we use simple reparameterizations of model                                                                                over hidden variables, which can be computed after
parameters that allow our optimizers to work with ar-                                                                                performing an E-step, as a crude estimate of the
bitrary values. For example, in the MoG model we                                                                                local missing information ratio. This entropy has
use a “softmax” parameterization of the mixing coef-                                                                                a natural interpretation as the uncertainty about
ﬁcients πi =     exp (γi )
                           , for covariance matrices to                                                                                missing information, and thus can serve as a guide
                                              exp (γj )                                                                                between switching regimes of EM and ECG. For many
be symmetric positive deﬁnite, we use the Choleski de-                                                                                models with discrete hidden variables this quantity is
composition (or log variances for diagonal covariance                                                                                quite easy to compute. In particular, we deﬁne the
matrices). In HMMs, we reparameterize probabilities                                                                                Normalized Entropy term:
via softmax functions as well.                                                                                ¯            N    M
                                                                                Ht = N −1M n i p(z = i|xn , Θt ) ln p(z = i|xn , Θt )

                                                                                with z being discrete hidden variable taking on

                                                                                M values, and N observed data vectors xn . We
                                                                   Log−Likelihood + Const

 1.5                      ΘECG

                                                                                now simply switch between EM and ECG based on
               Line Search                                                                  −0.005
                                                                                thresholding this quantity:

                                                                                Hybrid EM-ECG algorithm:
                                                                                Gene Sequence:
               EM                                                                            −0.02                              ATGCCGGAGGCCC ...
  −1.5    −1    −0.5      0       0.5   1     1.5    2
                                                         Θ                                               0   200            400             600       800         1000                                              ¯
                                                                                • Perform EM iterations, evaluating Ht after each
                                                         1                                                                        E−Steps
                                                                                • If Ht ≥ τ Thena Switch to ECG
Figure 2. Left panel illustrates why ECG may converge to
                                                                                • Perform ECG, evaluating Ht at the end of each
a better local optimum. Right panel displays the learn-
ing curves for EM and ECG algorithms of training fully                                                                                line search
                                                                                • If Ht < τ Then Switch back to EM
connected 7-state HMM to model human DNA sequences.
                                                                                • Exit at either phase IF:
Both algorithms started from the same initial parameter
values, but converged to the diﬀerent local optimum.                                                                                1. (L(Θt ) − L(Θt−1 ))/abs(L(Θt )) < tol OR
                                                                                2. t > Tmax
Of course, the choice of initial conditions is very im-                                                                                a
                                                                                We are near the optimum or in plateau region with
portant for the EM algorithm or for ECG. Since EM                                                                                high entropy
is based on optimizing a convex lower bound on the
likelihood, once EM is trapped in a poor basin of at-                                                                                As we will see from the experimental results, this sim-
traction, it can never ﬁnd a better local optimum. Al-                                                                                ple hybrid EM-ECG algorithm performs no worse, and
gorithms such as split and merge EM [13] were devel-                                                                                often far better than either EM or ECG.
       x 10

   0       EM−ECG                                                                                0                              EM−ECG
                                                                             EM                                                                                ECG
                                                                                ECG                                               EM
                 ECG                                                                       −0.02                                                                                −0.01

 −10                                    4

                                        0                                                                             Unstructured
                                        −5          −3             −1            1                                      Sequence: CEDBCEDDAC ...
 −20                                                                                −0.025
       0          100        200        300              400             500         600              0         50         100        150        200           250       300                          0                20      40        60    80        100   120   140   160   180

              EM−ECG / EM                                                                                EM−ECG / EM                                                                                0.05
                                                                                EM−ECG / EM
                                                                   ECG                                                                                ECG                                               0

  −5                                                                                −0.2

                                                                                Log−Likelihood + Const
 −10                                                                                −0.4                                                                                −0.1

 −15                                                                                −0.6                                                                                ECG

                                        −2                                                 −1.2
                                                                                Structured                                                                                −0.35

 −30                                                                                Sequence: AEABCDEAEABCDE...
                                              −2               0             2             −1.4
 −35                                                                                −0.45
       0                10         20              30                   40            50          0         5         10         15         20         25           30   35                                        0                10              20          30         40          50

Figure 3. Learning curves for ECG, EM-ECG, and EM algorithms, showing superior (upper panels) and inferior (lower
panels) performance of ECG under diﬀerent conditions for three models: MoG (left), HMM (middle), and Aggregate
Markov Models (right). The number of E-steps taken by either algorithm is shown on the horizontal axis, and log
likelihood is shown on the vertical axis. For ECG and EM-ECG, diamonds indicate the maximum of each line search,
and the zero-level likelihood corresponds to the converging point of the EM algorithm. The bottom panels use “well-
separated”, or “structured” data for which EM possesses Quasi-Newton convergence behavior. All models in this case
converge in 10-15 iterations with stopping criterion: [L(Θt+1 ) − L(Θt )]/abs(L(Θt+1 )) < 10−15 . The upper panels use
“overlapping”, “aliased”, or “unstructured” data for which proposed algorithms performs much better.
5. Experimental Results                                                                                the data is “well-separated” into distinct clusters and
We now present empirical results comparing the per-                                                                                another “not-well-separated” case in which the data
formance of EM, ECG, and hybrid EM-ECG for learn-                                                                                overlaps in one contiguous region. Figure 3 shows that
ing the parameters of three latent variable models:                                                                                ECG and Hybrid EM-ECG outperform standard EM
Mixtures of Gaussians (MoG), Hidden Markov Mod-                                                                                in the poorly separated cases. For the well-separated
els (HMM), and Aggregate Markov Models. In many                                                                                case, the hybrid EM-ECG algorithm never switches to
latent variable models, performing inference (E-step)                                                                                ECG due to the small normalized entropy term, and
is signiﬁcantly more expensive compared to either the                                                                                EM converges very quickly. This is predicted by our
parameter updates (M-step) or the line search over-                                                                                analysis: in the vicinity of the local optima Θ∗ the di-
head in the CG step of ECG. To compare the perfor-                                                                                rections of the vectors P (Θ) L (Θ) and (−S)−1 L (Θ)
mance of the algorithms, we therefore simply compare                                                                                become identical (ﬁg. 1), suggesting that EM will have
the number of E-steps each algorithm executes until its                                                                                Quasi-Newton convergence behavior.
convergence. We ﬁrst show results on synthetic data
                                                                                We then applied our algorithms to the training of Hid-
sets, whose properties we can control to verify certain
                                                                                den Markov Models (HMMs). Missing information in
aspects of our theoretical analysis. We also report em-
                                                                                this model is high when the observed data do not well
pirical results on several real world data sets, showing
                                                                                determine the underlying state sequence (given the pa-
that our algorithms do work well in practice. Though
                                                                                rameters). We therefore generated two data sets from
we show examples of single runs, we have conﬁrmed
                                                                                a 5-state HMM, with an alphabet size of 5 characters.
that the convergence results presented in all our ex-
                                                                                The ﬁrst data set (“aliased” sequences) was generated
periments do not vary signiﬁcantly for diﬀerent initial
                                                                                from a HMM where output parameters were set to uni-
parameter conditions. For all of the reported experi-
                                                                                form values plus some small noise. The second data
ments, we used tol = 10−8 and τ = 0.5.
                                                                                set (“very structured sequences”) was generated from
                                                                                a HMM with sparse transition and output matrices.
5.1. Synthetic Data Sets                                                                                For the ambiguous or aliased data, ECG and hybrid
First, consider a mixture of Gaussians (MoG) model.                                                                                EM-ECG outperform EM substantially. For the very
We considered two types of data sets, one in which                                                                                structured data, EM performs well and exhibits second
                                                                           x 10

                                 5−Component MOG                       1                             50−Component MOG                                                                                65−Component MOG
                                                                                EM−ECG                                                                                EM−ECG
                         EM−ECG / EM
                                                                                ECG                                EM            0.006

                               ECG                                                                                ECG

 −0.06                                                                                EM

  −0.1                                                                −5                                                                       −0.01
              0           50          100    150    200         250        0                  200       400      600       800         1000                             0               100           200           300      400   500         600
      x 10
                                     5−State HMM                                                          8−State HMM                                                                                10−State HMM
  2                                                                                EM−ECG               0.01

  0       EM−ECG                                                                                ECG
                               ECG                        EM                                                   ECG                EM                                0

                                 DNA Sequence:                                                         DNA Sequence:                                                                                DNA Sequence:
                                      ATGCCGGAGGCCC ...                                                     ATGCCGGAGGCCC ...                                                                                ATGCCGGAGGCCC ...
 −8                                                                   −0.03                                                                   −0.03
      0                 200          400    600    800         1000                0           200       400         600   800         1000                             0                 200               400           600      800               1000

  0.01                               3−Class AMM                       0.02                            6−Class AMM                                                          0.05
                                                                                9−Class AMM
                       EM−ECG                                                               EM−ECG                               EM                                                    EM−ECG                                             EM

                                                                                Log−Likelihood + Const
                                                          EM          −0.02

 −0.01                                                                                −0.05

 −0.02                                                                                −0.1

 −0.03                                                                                −0.15

 −0.04                                                                −0.12                                                                                −0.2

                       ECG                                            −0.14
                                                                                ECG                                                                                ECG                                               E−Steps
 −0.05                                                                                −0.25

Figure 4. Learning curves for ECG, EM-ECG, and EM algorithms, displaying convergence performance under diﬀerent
conditions for three models: MoG (upper), HMM (middle), and Aggregate Markov Models (bottom). The number of
E-steps taken by either algorithm is shown on the horizontal axis, and log likelihood is shown on the vertical axis. For
ECG and EM-ECG, diamonds indicate the maximum of each line search, and the zero-level likelihood corresponds to the
converging point of the EM algorithm. The number of learned clusters for MoG model were 5 (left), 50 (middle), and 65
(right). For HMM model, the number of states were 5 (left), 8 (middle), and 10 (right). The number of learned themes
for the AMM model were 3 (left), 6 (middle), and 9 (right).
order convergence in the vicinity of the local optimum.                                                          determine class labels, and on more structured data,
                                                                                in which the proportion of missing information is very
Finally, we experimented with Aggregate Markov
                                                                                small. ECG and hybrid EM-ECG are superior to EM
Models (AMMs) [11]. AMMs model deﬁne a discrete
                                                                                by at least a factor of two for ambiguous data; for
conditional probability table pij = p(y = j|x = i)
                                                                                structured data EM shows the expected Quasi-Newton
using a low rank approximation. In the context of
                                                                                convergence behavior.
n-gram models for word sequences, AMMs are class-
based bigram models in which the mapping from
words to classes is probabilistic. In particular, the                                                            5.2. Real World Data Sets
class-based bigram model predicts that word w1 is                                                                In our ﬁrst experiment, we cluster a set of 50,000
followed by word w2 with probability: P (w2 |w1 ) =                                                              8 × 8 greyscale pixel image patches4 using a mixture
   C                                                                                of Gaussians model. The patches were extracted from
   c=1 P (w2 |c)P (c|w1 ) with C being the total number                                                          768×512 natural images, described in [14] (see ﬁg 5 for
of classes. Here, the concept of missing information
                                                                                an example of a natural image, and sample patches).
corresponds to how well or poor a set of words de-
                                                                                To speed-up the experiments, the patch data was pro-
termine the class labels C based on the observation
                                                                                jected with PCA down to a 10-dimensional linear sub-
words that follow them. The right panels of ﬁgure
3 show training of a 2-class 50-state AMM model on                                                                    The data set used was the imlog data set publicly
ambiguous (aliased) data, in which words do not well                                                             available at ftp://hlab.phys.rug.nl/pub/samples/imlog
space and the mixing proportions and covariances of           ing a small number of “soft” classes (t): P (W |A) =
the model were held ﬁxed. The means were initial-                t=1 P (W |t)P (t|A). Once again, we observe that for
ized by performing K-means. We experimented with              this simple model, this data set has a large fraction of
mixtures having M=2 up to M=65 clusters.                      missing information. Figure 4 displays the convergence
                                                              of EM, ECG, and EM-ECG algorithms for T=3,6,9.
                                                              with hybrid EM-ECG and ECG having superior con-
                                                              vergence over EM.

                                                              6. Discussion and Conclusions
                                                              Although we have focused here on discrete latent vari-
                                                              ables, the ECG and hybrid algorithms can also be de-
                                                              rived for latent variable models with continuous hidden
Figure 5. An example of a natural image and some samples      variables. As an example ﬁgure 6 illustrates conver-
of 8 × 8 grey pixel image patches, used in the clustering     gence behaviour of the Probabilistic Principal Com-
experiment.                                                   ponent Analysis (PPCA) latent variable model[9, 12],
                                                              which has continuous rather than discrete hidden vari-
Figure 4 displays the convergence of EM, ECG,                 ables. Here the concept of missing information is re-
and Hybrid EM-EC algorithms for M=5, M=50 and                 lated to the ratios of the leading eigenvalues of the
M=65. The experimental results show that with fewer           sample covariance, which corresponds to the elliptic-
mixture components EM outperforms ECG, since the              ity of the distribution. For “low-rank” data with a
components generally model the data with fairly dis-          large ratio EM performs well; for nearly circular data
tinct, non-contiguous clusters. As the number of              ECG converges faster.6
mixtures components increases, clusters overlap in
contiguous regions and the normalized entropy term            In some degenerate cases, where the proportion of
grows, suggesting a relatively high proportion of the         missing information is very high, i.e. M (Θ∗ ) ap-
missing information. In this case ECG outperform EM           proaches identity, EM convergence can be exponen-
by several orders of magnitude. Hybrid EM-ECG al-             tially slow. Figure 6 illustrates such example for the
gorithm is never inferior to either EM or ECG (using          case of HMM training using almost random sequences.
our untuned setting of switching threshold τ = 0.5).          It takes about 7,000 iterations for ECG and EM-ECG
                                                              to converge to the ML estimate, whereas even after
Our second experiment consisted of training a fully           250,000 iterations EM is still only approaching the lo-
connected HMM to model DNA sequences. For the                 cal optimum.
training, we used publicly available ”GENIE gene
ﬁnding data set”, provided by UCSC and LBNL [5],              In this paper we have presented comparative analy-
that contains 793 unrelated human genomic DNA se-             sis of EM and direct optimization algorithms for la-
quences. We applied our diﬀerent algorithms on 66             tent variable models, and developed a theoretical con-
DNA sequences with length varying anywhere between            nection between these two approaches. We have also
200 to 3000 genes per sequence. The number of states          analyzed and determined conditions under which EM
ranged from M=5 to M=10 and all the parameter val-            algorithm can demonstrate local-gradient and Quasi-
ues were randomly initialized. Figure 4 shows the             Newton convergence behaviors. Our results extend
convergence of EM, ECG, and Hybrid EM-ECG algo-               those of Xu and Jordan[15] who analyzed the conver-
rithms for M=5,8,10. This data set contains very com-         gence properties of the EM algorithm in the special
plex structure which is not easily modeled by HMMs,           case of Gaussian mixtures, and apply to any exponen-
resulting in a very high proportion of missing informa-       tial family model.
tion. As a result, hybrid EM-ECG and ECG substan-             Motivated by these analyses, we have proposed an al-
tially outperform EM in terms of convergence.                 ternative hybrid optimization method that can signif-
In our last experiment, we applied Aggregate Markov           icantly outperform EM in many cases and is almost
Models to the data set consisting of 2,037 NIPS au-           never inferior. We tested the proposed algorithms by
thors and corresponding counts of the top 1,000 most             6
                                                                   The slow convergence of EM in PPCA is also true for
frequently used words of the NIPS conference proceed-         factor analysis and especially for linear dynamic systems.
ings, volumes 1 to 12.5 The goal was to model the             In these models, there is large amount of missing informa-
probability that an author A will use word W us-              tion due to the fact that latent variables are continuous
                                                              and they can be rotated without aﬀecting the likelihood as
     NIPS corpus used in the experiments is publicly avail-   long as the parameters are rotated accordingly.
able at http://www.cs.toronto.edu/∼roweis/data.html
  0.1                                                                                x 10
                    Probabilistic PCA                                          Probabilistic PCA                                                                          5−State HMM
                                                                                EM                                    0

                                               EM                                                                                EM−ECG /

                                                                                Log−Likelihood + Const
         ECG                                                                                ECG
 −0.1                                                                    ECG                                                                  −2
                             10                                                          3
                                                               −10                                                                            −4

 −0.3                                                          −15                                                                            −6
                              0                                                          0

                             −5                                                                                −8
 −0.4                                                          −20                      −2

                             −10                                                        −3
                              −10   −5   0      5   10                                         −2   0      2                                                                                E−Steps
                                                                                0            50,000   100,000   150,000    200,000   250,000
 −0.5                                                          −25
     0         50           100          150             200         0         50       100         150        200

Figure 6. Learning curves for ECG (dots) and EM (solid lines) algorithms, showing superior (left) and inferior (middle)
performance of ECG. The left panel uses ”ill-conditioned” data for which ECG converges quickly; the middle panel uses
“low-rank” data for which EM performs better. Right panel displays ”non-converging” case of the EM. Very unstructured
data (30 sequences, each of length 50) was generated from a full 5-state HMM with alphabet size of 5. Parameter values
were set to be uniform plus some small uniform noise. ECG and EM-ECG converge in about 7,000 iterations, whereas
after even 250,000 iterations, EM is only approaching to the ML estimate.
training several basic latent variable models on sev-                                          [5] GENIE gene data set. LBNL and UC Santa Cruz,
eral synthetic as well as real world data sets, reporting                                          http://www.fruitfly.org/sequence.
convergence behavior and explaining the results with                                           [6] Mortaza Jamshidian and Robert I. Jennrich. Accel-
reference to our analysis.                                                                         eration of the EM algorithm by using quasi-newton
                                                                                methods. J. of the RS Society series B, 49, 1997.
Our convergence analysis can also be extended to
a broader class of bound optimization techniques,                                              [7] Meng X. L. and van Dyk D. Fast EM-type imple-
                                                                                mentations for mixed eﬀects models. J. of the Royal
such as iterative scaling (IS) algorithms for parameter
                                                                                Statistical Society series B, 60:559–578, 1998.
estimation in maximum entropy models[2] and the
recent CCCP algorithm for minimizing the Bethe free                                            [8] Richard A. Redner and Homer F. Walker. Mixture
energy in approximate inference problems[16]. These                                                densities, maximum likelihood and the EM algorithm.
                                                                                SIAM Review, 26(2):195–239, April 1984.
analyses allow us to gain a deeper understanding of
the nature of these algorithms and the conditions                                              [9] S. T. Roweis. EM algorthms for PCA and SPCA.
under which certain optimization techniques can                                                    In Advances in neural information processing systems,
                                                                                volume 10, pages 626–632, Cambridge, MA, 1998.
be expected to outperform others. Based on these
extended analyses we are designing accelerated ﬁtting                                         [10] Ruslan Salakhutdinov. Relationship between gra-
algorithms for these models as well.                                                               dient and EM steps in latent variable models.
Acknowledgments                                                                               [11] Lawrence Saul and Fernando Pereira. Aggregate and
We would like to thank Yoshua Bengio and Drew Bag-                                                 mixed-order Markov models for statistical language
nell for many useful comments and Carl Rasmussen for                                               processing. In Proceedings of the Second Conference
                                                                                on Empirical Methods in Natural Language Process-
providing an initial version of conjugate gradient code.
                                                                                ing, pages 81–89. 1997.
References                                                                                [12] M. E. Tipping and C. M. Bishop. Mixtures of prob-
 [1] S.E. Atkinson. The performance of standard and hy-                                            abilistic principal component analysers. Neural Com-
     brid EM algorithms for ML estimates of the normal                                             putation, 11(2):443–482, 1999.
     mixture model with censoring. J. of Stat. Computa-                                       [13] Naonori Ueda, Ryohei Nakano, Zoubin Ghabramani,
     tion and Simulation, 44, 1992.                                                                and Geoﬀrey E. Hinton. SMEM algorithm for mixture
                                                                                models. Neural Computation, 12(9):2109–2128, 2000.
 [2] Stephen Della Pietra, Vincent J. Della Pietra, and
     John D. Laﬀerty. Inducing features of random ﬁelds.                                      [14] J. H. van Hateren and A. van der Schaaf. Independent
     IEEE Transactions on Pattern Analysis and Machine                                             component ﬁlters of natural images compared with
     Intelligence, 19(4):380–393, 1997.                                                            simple cells in primary visual cortex. In Proceedings
                                                                                of the Royal Society of London, pages 359–366, 1998.
 [3] A. P. Dempster, N. M. Laird, and D. B. Rubin. Max-
     imum likelihood from incomplete data via the EM al-                                      [15] L. Xu and M. I. Jordan. On convergence properties
     gorithm. J. of the RS Society series B, 39:1–38, 1977.                                        of the EM algorithm for Gaussian mixtures. Neural
                                                                                Computation, 8(1):129–151, 1996.
 [4] Chris Fraley. On computing the largest fraction of                                       [16] Alan Yuille and Anand Rangarajan. The convex-
     missing information for the EM algorithm and the                                              concave computational procedure (CCCP). In Ad-
     worst linear function for data augmentation. Tech-                                            vances in NIPS, volume 13, 2001.
     nical report, University of Washington.

To top


Gotham Gazette

The Place for New York Policy and politics

Citizens Union

Court Allows Man Who Falsely Confessed to Murder to Seek Money from the State

by Emily Jane Goodman, Apr 15, 2011
douglas warney
Photo New York State courts
Overturning lower courts, the state's highest bench issued a ruling that will allow Douglas Warney, who was convicted on the basis of what could have been a coerced confession, to seek financial compensation from the state.

Recently Gotham Gazette reported on the confession of Douglas Warney, who was convicted for a 1996 murder in Rochester but was later exonerated when DNA determined he was actually innocent.

Nevertheless, two New York courts found that he should not receive a state settlement compensating him for his wrongful conviction and the subsequent nine years he spent in prison. The court said his false confession constituted misconduct and so barred him from receiving compensation.

In late March, though, the state's highest court ruled in Warney's favor, reversing the lower courts' dismissal of his claim.

Warney's lawsuit seeking compensation had been brought in the Court of Claims, which hears cases against the state. In its ruling, the court is to rely on the Court of Claims Act and the Unjust Conviction and Imprisonment Act, which requires that a claimant seeking compensation had been convicted of a crime, sentenced to imprisonment and served at least part of the sentence. In addition, in order to receive a settlement, the claimant must have been pardoned on the ground of innocence or had a conviction that was reversed, vacated or dismissed.

The Court of Claims, however, rejected Warney's suit because language in the statute bars compensation for wrongful convictions that were caused by the defendant's own "misconduct." In other words, although DNA unequivocally proved that Warney is innocent, the confession he gave the police, which was used to convict him, constituted "misconduct" in the view of the Court of Claims judge, Renee Forgensi Minarik.

Allegations of Misconduct

The specific issue confronting the Court of Appeals was whether Minarik had improperly dismissed Warney's case without allowing the litigation to proceed further. The judge had decided that Warney's papers at the earliest stage of litigation should disclose in detail how his confession came about. The Court of Appeals opinion written by Judge Carmen Ciparick/a> agreed with Warney's counsel, Peter Neufeld, that Warney did not have to submit his evidentiary proof that early in the litigation proceedings.

Judge Robert Smith, who appeared particularly concerned when the case was argued in Albany, had posed many questions to the state's attorney, Alison Nathan, who is expected to become a federal judge. Smith asked how Warney could have confessed to specific facts of the crime that he couldn't have known both because of his limited mental abilities and because there is no question that he is innocent. Smith questioned whether the police interrogators gave Warney information, known only to them and the real perpetrator, that Warney then parroted back when asked about the murder.

At oral arguments and in his concurring opinion, Smith referred to the meaning of "coerced confession" and defined it as the sort of "calculated manipulation that appears present here." In general, though, Smith concurred with the majority opinion, which relied largely on procedural technicalities surrounding the dismissal. It did not give Warney's lawyers an opportunity to prove that it was not his acts that constituted misconduct. Smith feared that would lead to more general arguments against dismissal.

Criminal Knowledge

Warney, who had a history of making false police reports, has claimed that he had been denied access to a lawyer after police detained him for questioning in the murder. Warney eventually admitted to the murder in an unusually detailed confession -- one containing details that only the true perpetrator and the police could have known. But after he was convicted and sentenced to 25 years to life, Warney, who was then represented by the Innocence Project, claimed innocence. It was nine years later that DNA testing excluded him as a match. His conviction was reversed based on newly discovered evidence.

In ruling on Warney's request for compensation, lower courts said Warney had not offered convincing evidence that the police must have fed him detail of the crime. Ciparick's opinion reversed the lower courts , which had said that Warney would have had to present convincing evidence in the initial litigation on a possible settlement that the police fed him the details.

The appeals court determined that Warney did not have to prove his case in the lower courts but simply provide enough evidence to keep the case in court for trial. In other words, the Court of Appeals found it was premature for the Court of Claims judge to decide that she was not convinced by the facts Warney presented.

In his submission to the Court of Claims, Warney maintained that his conviction "was the direct result of the intentional and malicious actions of members of the [police] who fabricated and coerced a false confession from ... a man whom they knew had a history of serious mental health problems."

Referring in his opinion to some of the details in the confession, Smith asks, as the prosecutor had asked the trial jury, how a man whose innocence has been established could have known some of these facts he presented during police questioning.

"It is hard," Smith writes, "to imagine an answer other than that he learned them from the police. ... The police took advantage of Warney's mental frailties to manipulate him into giving a confession that contained seemingly powerful evidence corroborating its truthfulness -- when, in fact, the police knew, the corroboration was worthless. ... This sort of police conduct, if proved at trial would be sufficient to show that Warney 'did not by has own conduct cause or bring about his conviction.'''

Answering a question asked at oral argument, Smith said a false confession that was not coerced could be a barrier to compensation from the state. But, he wrote, "A confession cannot fairly be called 'uncoerced' that results from the sort of calculated manipulation that appears to be present here -- even if the police did not actually beat or torture the confessor, or threaten to do so. "

As clear as he appears to be about Warney, Smith is at least equally concerned with discouraging "frivolous claims." "This case is easy, because this claimant appears ... to have an exceptionally strong case," Smith wrote. "Our decision today should not be read as implying that any claimant can, by skillful pleading, get a significantly weaker case past a motion to dismiss."

According to Neufeld approximately 25 percent of those individuals who have been exonerated had confessed to crimes they did not commit. This week, Jeffrey Deskovic, who spent 15 years in prison after being convicted of raping, beating and strangling a high school classmate, but was later exonerated by DNA testing, settled his federal civil rights case against Westchester County for $6.5 million.

While it is not every day that cases such as Warney's and Deskovic's are brought, Neufeld hopes that the opinion in Warney's suit will have an effect on other local cases and in other states. That is, while false confessions are not uncommon, they should not generally prevent an exonerated individual from claiming compensation. Certainly as Smith wrote, the wrongfully convicted should have their day in court and an opportunity to establish their claim.

Other claims, though, will wait for another day. The next question would involve whether there will be a trial in Warney v State of New York. According to Peter Neufeld, there have been no discussions with the state about how the case will proceed now that the Court of Appeals has reversed the dismissal. It is likely the two sides will reach a monetary settlement.

Emily Jane Goodman is a New York State Supreme Court Justice

Editor's Choice


Chilean student march ends in arrests

IOL pic aug11 chile education protests Reuters Students throw stones at a riot police vehicle, which was set alight by a molotov cocktail, during an anti-government rally to demand changes in the public state education system, in Valparaiso, about 120km northwest of Santiago.

Santiago - Mass student protests against the government of Chilean President Sebastian Pinera left 396 people arrested and injured 55 police officers and 23 civilians, the authorities said on Wednesday.

Tens of thousands of people gathered across the country on Tuesday to demand free, good-quality public education. This was the largest protest in Chile since the end in 1990 of the military dictatorship led by Augusto Pinochet.

Around 100 000 people gathered in Santiago alone, where some masked people clashed with police and vandalised private property. The government complained that these were violent demonstrators, while student leaders denied any of their protesters had committed criminal acts.

Last week a similar protest left 874 people arrested and 90 police officers injured.

The Chilean government planned to make an updated proposal to students on Wednesday, and called for negotiations.

In Chile, most school education and all universities require monthly fees. Student protests started three months ago and have already led to a change of minister.

The ongoing protests have eroded Pinera's approval rating, which sank to 26 per cent according to an opinion poll published last week, the worst of any Chilean president since 1990. - Sapa-dpa

sign up


Mathematical and Computer Sciences Building

News & Announcements

Dr. Artem Zvavitch received his Ph.D. from Weizmann Institute of Science, Israel. Dr. Zvavitch is the current Graduate Coordinator. His primary research areas include Convex Geometry, Geometric Functional Analysis, and applications of Probability and Harmonic Analysis. Dr.
Zvavitch has been awarded almost $700,000 in NSF and other grants. Additionally, he has organized 20 various conferences and seminars, including the Informal Analysis Seminar, and is referee and/or reviewer for over 30 journals and publications.


Frequently Asked Questions

How do I access the library databases from off campus?

I was successfully logged into the databases but was suddenly blocked from continued use. What happened and what can I do to resume use?

What are Discovery Tools?

What is EBSCOhost Integrated Search?

What is WorldCat Local?

Where can I find particular call numbers in the library?

How do I access a wireless network connection in the library?

How do I access the library databases from off campus?

Users will use a SCCC User ID Login and Password, in fact, the same ID as your Angel login. When you attempt to use a Begley Library database off-campus using the EBSCO Integrated Search box, or by clicking on the database links available under the "Research" box on the home page, a window will appear with instructions for inputting your User ID in order to gain access. If you still have questions, please call 518-381-1487, or click on the "Troubleshooting" link at the bottom of the login page, and please include in your email, your Student ID number, and a description of your question or problem.

If after successfully logging in to our databases with your SCCC User ID, you are blocked from continued access with the following, or similar message: "File missing: - docs/suspended.htm." A transaction threshold has been reached by your account and any further use for the next six hours has been blocked. PLEASE CALL the Begley Library Reference Desk at 381-1239 during library hours, or note the problem with us via the Ask a Librarian service. After verifying that you are a legitimate SCCC patron, the librarian can request that the block be removed.

What are Discovery Tools?

They are technologies or applications that provide a single point of access to a broad range of library materials. Instead of searching separate databases and online catalogs for books and articles, discovery tools try to simplify the search process and eliminate the need to repeat the same search over and over in multiple library resources. Some tools even integrate other library technologies that allow a user to link through and request and/or obtain materials in the same session. These tools also often allow librarians and users to make limited customizations to their interface. For individual descriptions of the Discovery Tools Begley Library is offering, please see below.

EBSCOhost Integrated Search

EBSCO Integrated search is a federated search tool that allows users to simultaneously search EBSCO databases as well as other electronic resources, including those from other database aggregators, OPACs, and publisher packages. EBSCO Integrated Search can be accessed using the search box at the top of the library's home page. To search across databases, choose to search by keyword, title, or author. You can also filter your results to resources with full text available, to peer reviewed sources, or to sources available only in Begley's catalog or database collection. Once you have entered your search term and the appropriate filters, click "search" and a new tab will open in your browser displaying the search results. Use the search tools on the left of the page to narrow or broaden your search results by subject, date of publication, publisher, or the many other filters available, and click "update" to see the changes in the results list.

WorldCat Local

provides Web access to WorldCat, "the world's richest database for discovery of materials held in libraries." This tools differs from federated search engines, because it is a portal that provides access to materials beyond Begley Library. Millions of books and articles held worldwide become available via resource sharing programs such as Illiad. There are also links to social networking tools like Twitter, RSS feeds and blogs. Users can customize their own account's interface.

Map of Library Call Numbers


2029 A Near Miss Odyssey

Surface of Eros
Image Credit: NASA/Eros

A University of Michigan-led research team has discovered that for the first time in history, scientists will be able to observe how the Earth’s gravity will disrupt a massive asteroid’s spin.

Scientists predict a near-miss when Asteroid 99942 Apophis passes Earth in 2029. An asteroid flies this close to the planet only once every 1,300 years. The chance to study it will help scientists deal with the object should it threaten collision with Earth.

Only about three Earth diameters will separate Apophis and Earth when the 400-meter asteroid hurtles by Earth’s gravity, which will twist the object into a complex wobbling rotation. Such an occurrence has never been witnessed but could yield important clues to the interior of the sphere, according to a paper entitled, "Abrupt alteration of the spin state of asteroid 99942 Apophis (2004 MN4) during its 2029 Earth flyby," accepted for publication in the journal Icarus.

The team of scientists is led by U-M’s Daniel Scheeres, associate professor of aerospace engineering, and includes U-M’s Peter Washabaugh, associate professor of aerospace engineering.

Apophis is one of more than 600 known potentially hazardous asteroids and one of several that scientists hope to study more closely. In Apophis’ case, additional measurements are necessary because the 2029 flyby could be followed by frequent close approaches thereafter, or even a collision.

Scheeres said not only is it the closest asteroid flyby ever predicted in advance, but it could provide a birds-eye view of the asteroid’s "belly."

"In some sense it’s like a space science mission ‘for free’ in that something scientifically interesting will happen, it will be observable from Earth, and it can be predicted far in advance," Scheeres said.

If NASA places measuring equipment on the asteroid’s surface, scientists could for the first time study an asteroid’s interior, similar to how geologists study earthquakes to gain understanding of the Earth’s core, Scheeres said. Because the torque caused by the Earth’s gravitational pull will cause surface and interior disruption to Apophis, scientists have a unique opportunity to observe its otherwise inaccessible mechanical properties, Scheeres said. Throwing the asteroid off balance could also affect its orbit and how close it comes to Earth in future years.

Credit: Donald E. Davis

"Monitoring of this event telescopically and with devices placed on the asteroid’s surface could reveal the nature of its interior, and provide us insight into how to deal with it should it ever threaten collision," Scheeres said.

The asteroid will be visible in the night sky of Europe, Africa and Western Asia.

The asteroid was discovered late last year and initially scientists gave it a 1-in-300 chance of hitting the Earth on April 13, 2029. Subsequent analysis of new and archived pre-discovery images showed that Apophis won’t collide with Earth that day, but that later in 2035, 2036, and 2037 there remains a 1-in-6,250 chance that the asteroid could hit Earth, Scheeres said. Conversely, that’s a 99.98 percent chance that the asteroid will miss Earth.

The asteroid is relatively small, about the length of three football fields. If it hit it wouldn’t create wide-scale damage to the Earth, but would cause major damage at the impact site, Scheeres said.

The team of scientists also includes Lance Benner and Steve Ostro of NASA’s Jet Propulsion Laboratory, Alessandro Rossi of ISTI-CNR, Italy, and Francesco Marzari of the University of Padova, Italy.


Skip to main content
Report this ad

See also:

How To Kick Your Business Legacy Up A Notch

Every entrepreneur and business leader waits too long before really working on the legacy that he wants to leave to society and his family. They realize too late that they don’t really want to be remembered for how many hours they spent on airplanes, how many emails they produced, or even how much money they made for the business.

If you disappeared today, what would your legacy show? What have you done for others? If you are not thinking in these terms, you may be making a mistake as a leader. Bill Gates will probably be more remembered in fifty years for the his efforts and Foundation to save lives in developing countries, than Steve Jobs for his “insanely great” consumer technology advances.

I just finished a new book on this subject “Leading with Your Legacy In Mind,” by Andrew Thorn, PhD, business coach, and psychologist, who has worked on leadership strategies with business leaders at all levels. I espouse the “legacy continuum” that he outlines for every leader to reframe over time how their efforts should be spent, for purposes of kicking their legacy up a notch or two:

  1. From passion to purpose. “Just follow your passion” only takes us so far. Passion can ultimately blind us, while purpose reminds us of how we can connect our strengths with the people and environments that will most appreciate and benefit from our skills and abilities. It focuses us on what we can give instead of what we can get.

  2. From change to growth. Leaders often forget that change is hard for anyone. The only time we really like change if when we are acting as the change agents and inviting others to change. Growth, on the other hand, is most simply defined as change by natural development. Growth is natural for everyone, as a symbol of individual maturity.

  3. From goals to aspirations. Goals are generally connected to the near-term boundaries or limits that we wish to overcome and the actions that we must take to overcome them. Aspirations are more intensely connected to our deeper yearnings. When we factor in our aspirations to guide us, we begin to connect to what really gives us value in life.

  4. From balance to focus. Work/life balance is not a natural business goal. In fact, finding more balance may be impossible, due to the many daily emergencies and problems, but we can all find the time to fine-tune our focus. Focus gives us a sharpness of vision, and improves our understanding, to create a legacy that will endure the chaos of our busy life.

  5. From accepting to understanding. Acceptance embodies the idea that we must get to a place where we approve of something that we disagree with. Understanding is a higher attribute, because it allows us to hold on to what we value most, while at the same time showing a sympathetic and even an positive attitude toward another point of view.

  6. From discussion to dialogue. A discussion is a conversation that involves holding onto and defending our differences, seeking a winner. A dialogue provides an opportunity to explore the uncertainties that exist and the questions that are yet to be answered, with the potential of improving our relationships and benefiting from the collective wisdom.

  7. From listening to hearing. Listening skills are important, to focus first on noticing what is being said and what is not being said. To hear, we must actively and anxiously be willing to take action on what is being requested of us. Our legacy is strengthened when we demonstrate an ability to take action to make things better for all parties.

  8. From success to significance. Success is a count of favorable outcomes, which may or may not be significant. Significance will always be around longer than you will be around. It has a life of its own, inspiring someone else to make an impact, and nothing can stop it once it starts rolling. Finally, significance satisfies our deepest aspirations.

  9. From ambition to meaning. Ambition is our early career drive to prove our worth to others, to achieve recognition, often without regard for the sacrifices we are making. When we make the shift from ambition to meaning, we let our authentic self be our guide. Meaning is the personal fulfillment we enjoy as we grow through our own experiences.

  10. From growing older to growing whole. Growing older concerns most of us because it fills our mind with visions of what we are going to lose. Growing whole, on the other hand, is working aligned with purpose, less stress and anguish, and more time living than working. Growing whole involves celebrating by giving back and enjoying a real legacy.

It’s never too early to start working on the image you want to be remembered by, rather than the opportunistic default driven by short-term objectives and challenges. Legacy planning is nothing more than an exercise in using your time wisely. The average career is 117,000 hours of work. How many have you spent so far moving your own legacy up a few notches?

Report this ad


Visual Magnetics

Visual Magnetics Added to Material ConneXion’s Library of Innovation

Visual Magnetics has announced that its MagnaMedia has been selected by Material ConneXion to include in its library in New York City. Material ConneXion is made up of an international team of multidisciplinary experts that bridge the gap between science and design to create practical manufacturing solutions.

Material ConneXion’s subscription-based materials library is the world’s largest library of advanced, innovative, sustainable materials and processes in the world, with ever-growing physical libraries around the globe and an online database that provides immediate access to over 7,000 materials. Material ConneXion uses its materials intelligence to help companies innovate. Materials are chosen through a strict review process where between 30 and 40 new materials are independently juried into the library every month.

“Designers look to the Material ConneXion library for inspiration and to explore the most innovative, sustainable and exciting products that exist for a wide variety of applications,” said Joe Deetz, CEO of Visual Magnetics. “We are excited to be represented there, with 18 samples of Visual Magnetics MagnaMedia currently on display. Our products allow the creation of truly customizable, versatile wallcoverings and graphics, while simultaneously reducing the negative environmental impact commonly associated with these applications.”

Visual Magnetics creates 100% recyclable products, sources PVC-free base materials, and manufactures select media from 20-50% post-consumer plastics. The company has implemented a complete end-of-life process which encourages customers to return all retired MagnaMedia to Visual Magnetics to be recycled for use in steel and electricity production. Furthermore, Visual Magnetics significantly reduces the amount of material that end-users need to use and greatly increases the ability to reuse products instead of discarding them by enabling graphics to be easily layered and repositioned.


Why Read the Mystics?

Why should anyone bother reading the mystics?

Okay, some people would rather spend their lives reading People magazine or playing with their Xbox or simply trolling Ebay for that amazing find. So be it. One of the qualities of lavish grace is that it doesn’t force itself on anyone. So the first and perhaps most important answer to my question is simply, “Because you want to.” But having once established that interest (or desire) is present, still, why bother? Why not just read the Bible (or sacred texts from around the world), or sample the best of contemporary religious writing, or (actually a very worthy goal) moderate our religious study with exploration of science, technology, literature, philosophy, literature and the arts?

Okay, so it makes sense to incorporate the reading of mystical writings with an overall balanced intellectual and spiritual life. Fair enough. But mysticism still can be a wonderful focus for anyone interested in a deepening inner/spiritual life. Meanwhile, in the first sentence of this paragraph I’ve given you a clue to what I see as the two-fold reason why anyone can profit from getting to know the great works of the Christian mystics (or of the mystics of any contemplative tradition, for that matter).

So here’s the two-fold reason why we should read the mystics:

  1. For intellectual formation — to increase knowledge in terms of theology, history, or literature.
  2. For spiritual formation — to nurture one’s own faith and to aid in the process of cultivating an on-going sense of Divine love and presence in your life.

Both of these reading strategies are important, but for different reasons. For the sake of this discussion, I’m going to call the academic study of mysticism the critical approach, and the spiritual study of mysticism the devotional approach.

Critical reading of the mystics involves reading the text for the purpose of increasing knowledge. This may involve either the pursuit of personal intellectual development, or original research for the purpose of making a contribution to the ongoing expansion of human knowlege. When we study the mystics, we study a singular phenomenon in the history of human experience, a phenomenon with religious, social, political, and even evolutionary implications. But whether undertaken for one’s personal pleasure or to support a social or professional goal, critical reading always requires maintaining a distance from the text. The critical reader interrogates the text he or she is studying. Why does the writer say this? What does the writer mean? What does this text reveal about the writer, about the historical context, or about any other issue being researched? Critical reading treats the text as an object, and approaches it in terms of its usefulness. Does reading the text bring enjoyment, or insight, or understanding, or helps to solve a larger puzzle that is being researched? If the answer to any of these questions is yes, then the text passes the usefulness test. But if the answer is no, then frankly reading such a text would be a waste of time.

We can discover much through critical reading of the writings of the mystics. Again and again, the mystics testify to the possibility of believing deeply in the reality and accessibility of Divine love — and, perhaps more dramatically, in experiencing that very love. Mystics provide an optimistic approach to theology — a healthy corrective to the way that popular religion all too often ends up focusing on the negatives (the “thou shalt nots”). By stressing Divine love over the wrath of God, the testimony of the great contemplatives can help readers to discover spiritual possibilities that take us far beyond the “get saved and live a good life” theology that characterizes the contemporary religious mainstream (and that so many ex-Christians and non-Christians have so sensibly rejected). Of course, mystics have throughout Christian history spoken in accordance with the dominant religious language of their time — which is why mystics like Walter Hilton or Teresa of Avila often sound harsh and overly penitential to a third millennium reader. When we read the mystics with a critical distance, we can more easily avoid being ensnared by such language, remembering that the voice of any given mystic will necessarily reflect the cutural milieu of their day, thereby freeing us to discern what is truly universal in the literature of mysticism.

Devotional reading of the mystics takes us beyond the mere accumulation of knowledge. This method involves reading a text for the purpose of spiritual growth, of deepening our relationship with God and our life in Christ. To read a mystical work this way means to actually enter into the text as a meditative discipline, looking for ways to encounter the Divine Presence within the reading experience. Rather than reading from a critical distance, a devotional reader seeks to be as engaged with the text, as involved with it, as possible. Just a critical reading seeks distance from the text, so a devotional reading seeks intimacy within it. In devotional reading, the text asks questions of the reader. What does reading this book reveal about your beliefs? About your conscious or subconscious assumptions about the nature and character of God? How does this text challenge you as a spiritual person? What gift does this text bring you in terms of your interior growth? Is God using this book to speak to you? If so, what is God saying? Devotional reading treats the text as an ikon — a window through which the the uncreated light of the Holy Spirit may shine.

While any book with some measure of spiritual content can be read in either a critical or devotional manner, the writings of the great contemplatives are particularly rewarding to either reading strategy. Therefore, the richest way to unpack the treasures of the mystics is to practice both types of reading. The mystics certainly deserve to be questioned; their assumptions and unstated beliefs about God, the world, and spirituality ought to be examined critically, not only by scholars but by anyone seeking to understand how God has been experienced by people throughout history. For example, the harsh language of penitential theology needs to be clearly understood for what it is, not only for the pure acquisition of knowledge, but also to unlock the text for devotional purposes. In other words, the critical approach to mystical writings is never enough, since it fails to meet the writing in the way that the author originally intended it to be read. Mystics did not (and do not) write to keep graduate students busy — they wrote to help their readers to be transformed and renewed in the image and likeness of God, to prepare themselves for the miracle of the Beatific vision, whether in this life through supernatural intervention or more fully in the next.

So for any person interested in reading the great works of mysticism, the critical approach needs to be balanced by time spent reading in a contemplative way. The Benedictines call this method of reading lectio divina — literally, “divine reading.” Lectio divina involves not only reading a text prayerfully and slowly, allowing the text to speak to us and possibly reveal Divine love to us, it also fosters an environment where other forms of prayer may emerge. In fact, Benedictine spirituality suggests that lectio naturally leads to the spiritual practices of meditatio (meditation, or what we might call spiritual reflection), oratio (conversational prayer), and contemplatio (contemplative prayer).

To summarize my answer to the question, “why read the mystics?” For some, the sheer joy of learning about religious culture will be reason enough. However, to truly appreciate the gifts that the mystics have to give, we need to read in a spirit of openness and devotion. In that context, we read the mystics in order to deepen our relationship with the Divine.

This post is an expanded revision of an essay I wrote about ten years ago for the original “Website of Unknowing.”

The Feast of Saint Hildegard
One Great Mystic Speaks of Another
A Mini-Sabbatical...
John Cassian's Three Renunciations
  • http://www.archive.typepad.com/brothertadhg/ Brother Tadhg

    Great post and I concur. Thanks for writing it.

    Humbly, might I suggest a third reason for reading the mystics, and that of ‘strangeness’! Their way of looking at the world was different(and therefore ‘strange) to twenty-first century man/woman, and it causes us to anyalyse more deeply what they’re saying. Jesus used that type of thinking when he said, youve heard it said, but I say…’, and in answer to where are you mother, brothers etc, he spoke of his followers as being member of his family, which would cause any Jew of that day to sit up and listen intently, because it’s using words, phrases and thoughts in ways different to what is considered the norm.

    I can heartily recommend the works of Julian Of Norwich, the works of Meister Eckhart, and as regards lectio divina I can recommend a book, albeit 20 yrs old, called ‘Too Deep for Words: Rediscovering Lectio Divina’ by Thelma Hall (Author).

    Brother Tadhg

  • Peter

    Thanks, Carl, for this clear distinction between the two types of reading. I especially appreciate the difference between questioning the text (in critical reading) and submitting to let the text question you (in devotional reading). This honors the various faculties (“agents” as per M. Eckhart) of the human soul, leaving us free (because grace does not force itself on us) but keeping us properly accountable to the potential of what is available to all in our mystical experience. It is this second form of reading that makes the Bible and all forms of true spiritual writing the most valuable it can be for us. I am greatly looking forward to the Beatific Vision, and I ask, “Why wait? Why not get started now?!”
    Thanks again, Carl,


Proposed Legislation Targets Racial Profiling

U.S. Rep. Steve Cohen on Friday proposed legislation aimed at addressing racial disparities in the criminal justice system, WREG reports. The new laws would identify cases of racial profiling and penalize those responsible for it. Cohen is asking U.S. Attorneys to examine racial profiling in their areas and engage their communities to find ways to eliminate it. A panel of prosecutors, defenders, civil rights, and faith-based leaders would be asked for input. “Law enforcement officers do a great service and we need them so much but they need to spend their time based on actual probable cause and not racial stereotypes,” Cohen said.