My blogs reporting quantitative financial analysis, artificial intelligence for stock investment & trading, and latest progress in signal processing and machine learning

Saturday, February 25, 2012

My baby was born on last Saturday

The gift by the God:
(picture by cell-phone)

No doubt, my new life starts...

Thursday, February 9, 2012

Compressed Sensing Talks in ITA Workshop in San Diego- Part II (Thursday)

In my previous post I definitely missed some talks in this ITA.

Tomorrow (Thursday) there will be many interesting talks on compressed sensing:
8:50: On L0 search for low-rank matrix completion, by Wei Dai, Imperial College London, Ely Kerman, UIUC, Olgica Milenkovic, UIUC

9:10 Orthogonal matching pursuit with replacement, by Inderjit Dhillon, University Of Texas, Prateek Jain, Microsoft, Ambuj Tewari, University Of Texas

3:00 Sparse sampling: bounds and applications, by Martin Vetterli, EPFL

4:15: Bilinear generalized approximate message passing (BiG-AMP) for matrix recovery problems Phil Schniter, Ohio State, Volkan Cevher, EPFL

There is another talk at the same time:
Construction of low-coherence frames using group theory, by Babak Hassibi, Caltech, Matthew Thill, Caltech

4:35:  Sparse recovery with graph constraints, by Meng Wang, Cornell, Weiyu Xu, Cornell, Enrique Mallada, Cornell, Kevin Tang, Cornell

4:55: Asymptotic analysis of complex LASSO via complex approximate message passing, by Arian Maleki, Rice, Laura Anitori, TNO, Netherlands, Zai Yang, Nanyang Technological University, Richard Baraniuk, Rice

In addition to the compressed sensing talks, there are many interesting talks on Music Information Retrieval, Clustering, Learning Theory, Graphical Models and Inference, and Statistical Machine learning & Applications.

Thursday will be a wonderful day.

Monday, February 6, 2012

A New Paper: Evolving Signal Processing for Brain-Computer Interface

We have a survey paper on BCI recently accepted by Proceedings of the IEEE (Special 100th Anniversary Issue):

Scott Makeig, Christian Kothe, Tim Mullen, Nima Bigdely-Shamlo, Zhilin Zhang, Kenneth Kreutz-Delgado, Evolving Signal Processing for Brain-Computer Interface, Proceedings of the IEEE, 2012

The paper surveys the past, the present, and the future of signal processing and machine learning in the cognitive state assessment especially BCI, wireless EEG, and mobile EEG.

The paper can be downloaded from here

Here is abstract:
Because of the increasing portability and wearability of noninvasive electrophysiological systems that record and process electrical signals from the human brain, automated systems for assessing changes in user cognitive state, intent, and response to events are of increasing interest. Brain-computer interface (BCI) systems can make use of such knowledge to deliver relevant feedback to the user or to an observer, or within a human-machine system to increase safety and enhance overall performance. Building robust and useful BCI models from accumulated biological knowledge and available data is a major challenge, as are technical problems associated with incorporating multimodal physiological, behavioral, and contextual data that may in future be increasingly ubiquitous. While performance of current BCI modeling methods is slowly increasing, current performance levels do not yet support widespread uses. Here we discuss the current neuroscientific questions and data processing challenges facing BCI designers and outline some promising current and future directions to address them.

Friday, February 3, 2012

Compressed Sensing Talks in ITA Workshop in San Diego (Sunday 2/5 - Friday 2/10)

From this Sunday we will have a great annual academic even in San Diego: ITA Workshop. Each year, the workshop invites many well-established scholars in the field of compressed sensing to give talks.

Here is the workshop calendar: http://ita.ucsd.edu/workshop/12/talks

Particularly, I found the following talks on compressed sensing/sparse signal recovery (I probably missed some):

Monday:
11:20: Quick partial sparse support recovery by Vincent Poor, Princeton, Ali Tajer, Princeton
3:00:  Information-theoretically optimal compressed sensing via spatial coupling and approximate message passing by David Donoho, Stanford, Adel Javanmard, Stanford, Andrea Montanari, Stanford

Thursday (I missed some interesting talks in this day. More complete list can be seen here: http://marchonscience.blogspot.com/2012/02/compressed-sensing-talks-in-ita_09.html):
8:50: On L0 search for low-rank matrix completion, by Wei Dai, Imperial College London, Ely Kerman, UIUC, Olgica Milenkovic, UIUC
9:10: Orthogonal matching pursuit with replacement, by Inderjit Dhillon, University Of Texas, Prateek Jain, Microsoft, Ambuj Tewari, University Of Texas
3:00: Sparse sampling: bounds and applications by Martin Vetterli, EPFL
3:40: Compressive depth acquisition cameras: Principles and demonstrations by Vivek Goyal, MIT
4:15: Construction of low-coherence frames using group theory by Babak Hassibi, Caltech, Matthew Thill, Caltech
4:15: Bilinear generalized approximate message passing (BiG-AMP) for matrix recovery problems Phil Schniter, Ohio State, Volkan Cevher, EPFL
4:35: Sparse recovery with graph constraints by Meng Wang, Cornell, Weiyu Xu, Cornell, Enrique Mallada, Cornell, Kevin Tang, Cornell
4:55:  Asymptotic analysis of complex LASSO via complex approximate message passing by Arian Maleki, Rice, Laura Anitori, TNO, Netherlands, Zai Yang, Nanyang Technological University, Richard Baraniuk, Rice

Friday:
11:20: Faster algorithms for sparse fourier transform, by Haitham Hassanieh, MIT, Piotr Indyk, MIT, Dina Katabi, MIT, Eric Price, MIT
Compressive sensing meets group testing: LP decoding for non-linear (disjunctive) measurements, by Chun Lam Chan, CUHK, Sidharth Jaggi, CUHK, Venkatesh Saligrama, BU, Samar Agnihotri, CUHK
1:35: The Big Data bootstrap, by Ariel Kleiner, UC Berkeley, Ameet Talwalkar, UC Berkeley, Purna Sarkar, UC Berkeley, Michael Jordan, UC Berkeley

In addition to these talks, there are other interesting talks on high-dimensional data analysis, information theory, and neuroscience/AI.

Next week should be a wonderful week, except an unhappy thing: this year ITA will be hold in a hotel in San Diego, not in UCSD campus as previous years. It's so inconvenient :(

Thursday, January 26, 2012

Andrew Ng: Machine learning and AI via large scale brain simulations

The location is changed to: CALIT2 ~ Atkinson Hall Auditorium
Time: Monday, January 30th, 2012, 11:00 am

Abstract

By building large-scale simulations of cortical (brain) computations, can
we enable revolutionary progress in AI and machine learning? Machine
learning often works very well, but can be a lot of work to apply because
it requires spending a long time engineering the input representation (or
"features") for each specific problem. This is true for machine learning
applications in vision, audio, text/NLP and other problems.
To address this, researchers have recently developed "unsupervised feature
learning" and "deep learning" algorithms that can automatically learn
feature representations from unlabeled data, thus bypassing much of this
time-consuming engineering. Many of these algorithms are developed using
simple simulations of cortical (brain) computations, and build on such
ideas as sparse coding and deep belief networks. By doing so, they exploit
large amounts of unlabeled data (which is cheap and easy to obtain) to
learn a good feature representation. These methods have also surpassed the
previous state-of-the-art on a number of problems in vision, audio, and
text. In this talk, I describe some of the key ideas behind unsupervised
feature learning and deep learning, and present a few algorithms. I also
speculate on how large-scale brain simulations may enable us to make
significant progress in machine learning and AI, especially perception.
This talk will be broadly accessible, and will not assume a machine
learning background.

Tuesday, January 24, 2012

Literature-CS: Sparse Signal Recovery/Compressed Sensing of ICASSP 2012

ICASSP 2012 has posted the technical program: http://www.icassp2012.com/RegularProgram.asp

Here are the sections on sparse signal recovery/compressed sensing:


SPTM-P6: Joint SPTM/SPCOM Session: Sampling Sparsity and Reconstruction II

SPCOM-P2: Sampling, Coding and Modulation

SPTM-L3: Compressed Sensing and Sparsity I

SPTM-L4: Compressed Sensing and Sparsity II

SPTM-L5: Compressed Sensing and Sparsity III

SPCOM-L4: Sparse Signal Processing for Communications and Networking

SPTM-P9: Sampling and Reconstruction

SAM-P5: Joint SAM/SPTM Session: Compressed Sensing and Sparse Signal Modeling


My paper will be presented at the section: SPTM-L4: Compressed Sensing and Sparsity II

The title is:
Z.Zhang, B.D.Rao, Recovery of Block Sparse Signals Using the Framework of Block Sparse Bayesian Learning

You can read it now from my website: http://sccn.ucsd.edu/%7Ezhang/Zhang_ICASSP2012.pdf
Codes can be downloaded at: http://sccn.ucsd.edu/%7Ezhang/BSBL_EM_Code.zip

The paper is the early work of the journal version:

Z.Zhang, B.D.Rao, Extension of SBL Algorithms for the Recovery of Block Sparse Signals with Intra-Block Correlation

The paper can be obtain from: http://arxiv.org/abs/1201.0862

Tuesday, January 10, 2012

A New Paper: Extension of SBL Algorithms for the Recovery of Block Sparse Signals with Intra-Block Correlation

We just finished a paper on block sparse model, which considers to exploit intra-block correlation with known or unknown block partition:

Zhilin Zhang, Bhaskar D. Rao , Extension of SBL Algorithms for the Recovery of Block Sparse Signals with Intra-Block Correlation, submitted to IEEE Transaction on Signal Processing, January 2012

The associated codes can be downloaded here: https://sites.google.com/site/researchbyzhang/bsbl

Here is the abstract:

We examine the recovery of block sparse signals and extend the framework in two important directions; one by exploiting intra-block correlation and the other by generalizing the block structure. We propose two families of algorithms based on the framework of block sparse Bayesian learning (bSBL). One family, directly derived from the bSBL framework, requires knowledge of the block partition. Another family, derived from an expanded bSBL framework, is based on a weaker assumption about the a priori information of the block structure, and can be used in the cases when block partition, block size, block sparsity are all unknown. Using these algorithms we show that exploiting intra-block correlation is very helpful to improve recovery performance. These algorithms also shed light on how to modify existing algorithms or design new ones to exploit such correlation for improved performance.

The paper can be downloaded here: http://arxiv.org/abs/1201.0862. The codes will be posted soon. But you can send emails to me for these codes right now.

In this paper, we proposed three algorithms (BSBL-EM, BSBL-BO, BSBL-L1) for the block sparse model when block partition is known, and three algorithms (EBSBL-EM, EBSBL-BO, EBSBL-L1) for the model when block partition is unknown.

Here are some highlights:

[1] These algorithms have the best recovery performance among all the existing algorithms

I have spent more than one month to read published algorithms, downloaded their codes, performed experiments, and sending emails to authors to ask for optimal tuning of parameters, etc. I did't find any existing algorithms have better performance than mine. If you find one, please let me know. 


Here is a comparison among all well-known algorithms when block partition is given (signal length was fixed while we changed the measurement number; see the paper for details) :

Here is a comparison among existing algorithms when block partition is unknown (signal length, measurement number, and the number of nonzero elements in the signal were fixed while we changed the nonzero block number; each block had random size and location. See the paper for details)



[2] These algorithms are the first algorithms that adaptively exploit intra-block correlation, i.e. the correlation among elements of a block.


[3] We revealed that intra-block correlation, if exploited, can significantly improve recovery performance and reduce the number of measurements.

Here is an experiment result showing our algorithms have better performance when intra-block correlation increases (see the paper for details)

[4] We also found that the intra-block correlation has little effects on the performance of existing algorithms. This is different to our finding on the MMV model, where we found temporal correlation has obvious negative effects on the performance of existing algorithms (for temporal correlation on the algorithm performance, see here).

Here is an experiment result showing the performance of Block-CoSaMP and Block-OMP is almost not affected by the intra-block correlation (see the paper for details).

Tuesday, November 8, 2011

Updated T-MSBL code

I just now updated the T-MSBL/T-SBL code. So, using the updated version, you need NOT to consider the tuning of parameters for a general compressed sensing problem. By a general compressed sensing problem, I mean the columns of the matrix A has unit L2-norm. When your problem does not satisfy this, you can first transform your original problem:
Y = A X + V
to
Y = A W W^{-1} X + V  = A' X' + V
such that A' has unit-norm columns. Once you obtain the result, you can obtain X by X = W X'.


 The calling of T-MSBL is easy:


o   When noise is large (e.g. SNR <=6 dB)
X_est = TMSBL(A, Y, 'noise', 'large')

o   When noise is mild (e.g. 7 dB <= SNR <=22 dB)
X_est = TMSBL(A, Y, 'noise', 'mild')

o   When noise is small (e.g. SNR >22 dB)
X_est = TMSBL(A, Y, 'noise', 'small')

o   When no noise
                                     X_est = TMSBL(A, Y, 'noise', 'no')

But note that the above number 6dB or 22dB is not an exact value. The two values just give you a rough concept of what is the 'small noisy case', what is the 'mild noisy case', and what is the 'strongly noisy case'.
In this sense, this does not mean T-MSBL requires to know the noise level.

When you use T-MSBL in some practical problems when you really have no idea what is the range of noise strength (such as gene feature extraction), simply use the calling corresponding to the 'mild noise case', i.e.
X_est = TMSBL(A, Y, 'noise', 'mild') 


I will update the code in the near future, such that in any case(noisy, noiseless, real variable, complex variable, large-scale data or small-scale data)  you only need to input X_est = TMSBL(A,Y). But I currently am very busy on my on-going papers (four journal papers in three fields), so please forgive me that I cannot do this now.







Friday, November 4, 2011

Minisymposium on New Dimensions in Brain-Machine Interfaces at UCSD

Wednesday, November 9, 2011
1pm-6pm
Fung Auditorium
Powell-Focht Bioengineering Hall
UC San Diego

The minisymposium highlights latest advances and emerging directions in
brain-machine and neuron-silicon interface technology and their
applications to neuroscience and neuroengineering.  Topics include
high-dimensional EEG and ECoG systems, wireless and unobtrusive
brain-machine interfaces, flexible bioelectronics, real-time decoding of
brain and motor activity, and signal processing methods for intelligent
human-system interfaces.


PROGRAM

1:00-1:10pm    Welcome

1:10-1:50pm    Engineering hope with biomimetic systems
              Wentai Liu, UC Santa Cruz

1:50-2:30pm    A low power system-on-chip design for real-time ICA based BCI applications
              Wai-Chi Fang, National Chiao-Tung University, Taiwan

2:30-3:10pm    Developing practical non-contact EEG electrodes
              Yu Mike Chi, Cognionics

3:10-3:50pm    A new platform for BCI: from iBrain to the Stephen Hawking project
              Philip Low, Neurovigil


3:50-4:20pm    Coffee break


4:20-5:00pm    Interdisciplinary approaches to design high performance brain-machine interfaces
              Todd P. Coleman, UC San Diego

5:00-5:40pm    Evolving data collection and signal processing methods for intelligent human-system interfaces
              Scott Makeig, UC San Diego

5:40-6:00pm    Panel discussion


Organized by:

Tzyy-Ping Jung <tpjung@ucsd.edu>
Center for Advanced Neurological Monitoring,
Institute of Engineering in Medicine <http://iem.ucsd.edu>, and
Institute for Neural Computation <http://inc.ucsd.edu>

With support from:

Qualcomm <http://www.qualcomm.com>, and
Brain Corporation <http://www.braincorporation.com>

Monday, October 31, 2011

Compressed Sensing Work by My Friends and Colleagues

Recently some of my friends and colleagues sent me their recent work on compressed sensing/sparse signal recovery. Thanks them for keeping me informed! Here are their nice work: I welcome everybody send me his/her work and I would like to introduce his/her work in my blog :)

Hakan informed me of his work:
Karahanoglu, N.B., and Erdogan, H., “Compressed sensing signal recovery via A* Orthogonal Matching Pursuit,” ICASSP’11, Prag, May 2011.
 The journal version is:
Karahanoglu, N.B., and Erdogan, H., “A* orthogonal matching pursuit: best-first search for compressed sensing signal recovery,” submitted, available as: arxiv 1009.0396,  last update in Sep. 2011.
 Matlab code can be downloaded here

The abstract reads:

Compressed sensing aims at reconstruction of sparse signals following acquisition in reduced dimensions, which makes the recovery process under-determined. Due to sparsity, required solution becomes the one with minimum â„“0 norm, which is untractable to solve for. Commonly used reconstruction techniques include â„“1 norm minimization and greedy algorithms. This manuscript proposes a novel semi-greedy approach, namely A* Orthogonal Matching Pursuit (A*OMP), which performs A* search for the sparsest solution on a tree whose paths grow similar to the Orthogonal Matching Pursuit (OMP) algorithm. Paths on the tree are evaluated according to an auxiliary cost function, which should compensate for different path lengths. For this purpose, we suggest three different structures. We show that novel dynamic cost functions provide improved results as compared to a conventional choice. Finally, we provide reconstruction results on both synthetically generated data and images showing that A*OMP outperforms well-known CS reconstruction methods, Basis Pursuit (BP), OMP and Subspace Pursuit (SP).


Kiryung informed me of his latest updated work:

Kiryung Lee, Yoram Bresler, Marius Junge, Subspace Methods for Joint Sparse RecoveryarXiv:1004.3071v4

The abstract reads:
We propose robust and efficient algorithms for the joint sparse recovery problem in compressed sensing, which simultaneously recover the supports of jointly sparse signals from their multiple measurement vectors obtained through a common sensing matrix. In a favorable situation, the unknown matrix, which consists of the jointly sparse signals, has linearly independent nonzero rows. In this case, the MUSIC (MUltiple SIgnal Classification) algorithm, originally proposed by Schmidt for the direction of arrival problem in sensor array processing and later proposed and analyzed for joint sparse recovery by Feng and Bresler, provides a guarantee with the minimum number of measurements. We focus instead on the unfavorable but practically significant case of rank-defect or ill-conditioning. This situation arises with limited number of measurement vectors, or with highly correlated signal components. In this case MUSIC fails, and in practice none of the existing methods can consistently approach the fundamental limit. We propose subspace-augmented MUSIC (SA-MUSIC), which improves on MUSIC so that the support is reliably recovered under such unfavorable conditions. Combined with subspace-based greedy algorithms also proposed and analyzed in this paper, SA-MUSIC provides a computationally efficient algorithm with a performance guarantee. The performance guarantees are given in terms of a version of restricted isometry property. In particular, we also present a non-asymptotic perturbation analysis of the signal subspace estimation that has been missing in the previous study of MUSIC.

This is the fourth version. I read its third version, which has about 30 pages. However, the fourth version doubles the page number. So I asked Kiryung what are the main changes compared to the previous version. Kiryung replied:

"We added another subspace greedy algorithm for partial recovery step. This ends up with better empirical performance.  All algorithms presented in this paper have guarantees.  We updated the analysis by using a version of RIP,  which is different from the original uniform RIP and is satisfied by a weaker condition.  "



Justin sent me his journal paper on MMV model using AMP. It's a very cool algorithm. However, the journal paper has not been opened to the public. But I think you can read his conference paper soon:

J. Ziniel and P. Schniter, ``Efficient Message Passing-Based Inference in the Multiple Measurement Vector Problem,'' to appear in Proc. Asilomar Conf. on Signals, Systems, and Computers (Pacific Grove, CA), Nov. 2011.




-------------------------------------
Image: Nepenthes. hamata grown in my patio.





Friday, October 28, 2011

Call for Paper: Special Issue on Dependent Component Analysis

There will be a special issue on Dependent Component Analysis in EURASIP Journal on Advances in Signal Processing.  Dependent component analysis (DCA) is a big extension of ICA, and is one of the main directions of the ICA field in recent years.  I think this issue should be a good window to see current progress on DCA.

The issue includes the following topics (but not limited to):
- Multidimensional component analysis
- (Independent) subspace analysis
- Vector component analysis
- Correlated component analysis
- Topographical component analysis
- Tree-dependent component analysis
- Blind dependent component analysis
- Informed (Bayesian) dependent component analysis
- and their applications

Manuscripts Submission Date:  Feb 1, 2012.
Publication date: Oct.1, 2012.


--------------------------------------------------------
Image: Nepenthes.talangensis, grown in my partio.

Thursday, October 20, 2011

How people in science see each other

Today Tobias sent us a picture titled "How people in science see each other". It is very funny.  Enjoying! (click the picture for larger view)

Noise Folding Puts Tough Requirements on Compressed Sensing Algorithms?

Tonight I read two papers on noise folding in the compressed sensing problem. They are:

[1] M.A.Davenport, J.N.Laska, J.R.Treichler, R.G.Baraniuk, The Pros and Cons of Compressive Sensing for Wideband Signal Acquisition: Noise Folding vs. Dynamic Range. Preprint, April, 2011


The noise folding is a silent topic in the hot compressed sensing field. The problem is described as follows:
y = A (x + n) + v                 (1)
Namely, the signal itself contains noise (called 'signal noise' or 'source noise'). v is the measurement noise. The model can be rewritten as
y = Ax + (An+v) = Ax + u.   (2)
Intuitively, the 'noise' has increased, and this will bring "troubles" to algorithms.

The above two papers rigorously analyze how the signal noise n affects the estimation quality, the RIP condition, and so on.

In [1], the authors consider the case when the matrix A is orthonormal and v is not present. They found that the noise folding has a significant impact on the amount of noise present in CS measurements; every time one doubles the subsampling factor g (i.e. the ratio of column number of A to its row number), the SNR loss increases roughly by 3dB, namely,

In [2] the authors considered a general case (i.e. A is not necessarily orthonormal and the measurement noise v is present) and showed that the model (1) is equivalent to
y_hat = Bx +z
where B is a matrix whose coherence and RIP constants are very close to those of A, and z is zero-mean white noise vector with covariance matrix (sigma^2 + g * sigma_0^2) I., where E{v} = 0, E{v v^T} = sigma^2 * I,  and E{n}=0, E{n n^T} = sigma_0^2 * I.. The result also suggests that the effect of signal noise n is to degrade the SNR by a factor of g.

Clearly, these results tell every algorithm designer (note: most algorithms essentially perform on the rewritten model (2) ):

1) To design algorithms that work well under strong noise environment, especially when the subsampling factor g is large.

2) To design algorithms that do not need any prior knowledge on the noise level. In practice we perhaps can get some knowledge on the strength of measurement noise v, but we have much less knowledge on the strength of signal noise n. Consequently, we don't know the strength of the equivalent noise u (see the rewritten model (2)). Note that many algorithms use the knowledge of the noise level (in model (2) ) to set a good value for their regularization parameter. So, this means that the regularization parameter (related to the noise level) should be automatically learned by algorithms themselves, not pre-set.

A quick example is the EEG source localization. In this application, the measurement noise level is well controlled by  EEG recording machines. However, we know little about the strength of the signal noise. As we have known,  the regularization parameter strongly affects algorithms' performance. So, an algorithm with user-defined regularization parameter may perform far from optimally.

Wednesday, October 19, 2011

Neuroskeptic: What Is Brain "Activation" on fMRI?

Neuroskeptic has a blog entry, reporting a 2010 paper:


which argues that 80% of the BOLD signal is caused by internal processing of neurons, and only 20% is due to input from other neurons.

This result again points out the big gap between fMRI activity and EEG activity, since the input from other neurons is thought to be the "source" of EEG.  This also gives us a caution on a group of EEG source localization approaches which use fMRI activity as a spatial constraint for the localization problem. 


The abstract of the paper is:

An important constraint on how hemodynamic neuroimaging signals such as fMRI can be interpreted in terms of the underlying evoked activity is an understanding of neurovascular coupling mechanisms that actually generate hemodynamic responses. The predominant view at present is that the hemodynamic response is most correlated with synaptic input and subsequent neural processing rather than spiking output. It is still not clear whether input or processing is more important in the generation of hemodynamics responses. In order to investigate this we measured the hemodynamic and neural responses to electrical whisker pad stimuli in rat whisker barrel somatosensory cortex both before and after the local cortical injections of the GABAA agonist muscimol. Muscimol would not be expected to affect the thalamocortical input into the cortex but would inhibit subsequent intra-cortical processing. Pre-muscimol infusion whisker stimuli elicited the expected neural and accompanying hemodynamic responses to that reported previously. Following infusion of muscimol, although the temporal profile of neural responses to each pulse of the stimulus train was similar, the average response was reduced in magnitude by ∼79% compared to that elicited pre-infusion. The whisker-evoked hemodynamic responses were reduced by a commensurate magnitude suggesting that, although the neurovascular coupling relationships were similar for synaptic input as well as for cortical processing, the magnitude of the overall response is dominated by processing rather than from that produced from the thalamocortical input alone.