Tamir Hazan, Ph.D.
Faculty of Industrial Engineering and Management
Deep learning revolutionzed AI and machine learning techniques can be used to achieve human-like behavior. We explore attention models that allow to interprate and improve the prediction process of a learner.
Relevant materials: High Order Attention Models for Visual Question Answering, Factor Graph Attention, Audio-Visual Scene-Aware Dialog.
|visual question answering||factor graph attention||scene aware dialog|
Markov random fields, convex duality and message-passing: To efficiently predict outcomes in complex systems one can use graphical models and structured predictors. In this setting each predictor provides partial outcomes, e.g., the semantic labels of a region in an image, and global consistency for the structured prediction is maintained by passing messages between these regions. These concepts, emerging from Judea Pearl’s belief propagation algorithm, can be interpreted in terms of optimization theory. Applying Fenchel duality we develop the convex norm-product belief propagation, and its high-oder extensions, which enforce consistency between overlapping predictors using dual block coordinate descent. This provides us with the means to use cloud computing platforms to distribute and parallelize the prediction while maintaining consistency between its subproblems (code available, also for Amazon EC2). Estimating the parameters of region based predictors increase their accuracy in many real-life programs, and currently we achieve with these methods state-of-the-art results in various computer vision applications.