信息过滤 和 时间检测

最近主要的工作是信息过滤和新闻的时间检测。

新闻的时间检测比较容易，主要是考虑的情况要全面，别的就没有什么了。

信息过滤还是比较复杂。首先要过滤掉色情的词。然后对于政治的东西，需要检测出文章的感情色彩，是对当前政府持否定态度的还是肯定的。最后还要对那种纯粹骂人的文章过滤掉。过滤的目的一方面是符合法律的要求，另一方面是大家的时间都宝贵，没空听你胡说八道。

WWW 2007 论文

Track: Browsers and User Interfaces

• Session: Personalization
• Homepage Live: Automatic Block Tracing for Web Personalization
• Open User Profiles for Adaptive News Systems: Help or Harm?
• Investigating Behavioral Variability in Web Search
• Session: Smarter Browsing
• CSurf: A Context-Driven Non-Visual Web-Browser
• GeoTracker: Geospatial and Temporal RSS Navigation
• Learning Information Intent via Observation

Track: Data Mining

• Session: Mining in Social Networks
• Wherefore Art Thou R3579X? Anonymized Social Networks Hidden Patterns and Structural Steganography
• Information Flow Modeling based on Diffusion Rate for Prediction and Ranking
• NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks
• Session: Mining Textual Data
• Summarizing Email Conversations with Clue Words
• Organizing and Searching the World Wide Web of Facts - Step Two: Harnessing the Wisdom of the Crowds
• Do Not Crawl in the DUST: Different URLs with Similar Text
• Session: Predictive Modeling of Web Users
• Demographic Prediction based on User's Browsing Behavior
• Why We Search? Visualizing and Predicting User Behavior
• Topic Sentiment Mixture: Modeling Facets and Opinions in Weblogs
• Session: Identifying Structure in Web Pages
• Page-level Template Detection via Isotonic Smoothing
• Towards Domain Independent Information Extraction from Web Tables
• Web Object Retrieval
• Session: Similarity Search
• New Suffix Tree Similarity Measure for Document Clustering
• Scaling Up All-Pairs Similarity Search
• Detecting Near-Duplicates for Web Crawling

Track: E* Applications

• Session: E-Communities
• The Complex Dynamics of Collaborative Tagging
• Expertise Networks in Online Communities: Structure and Algorithms
• Internet-Scale Collection of Human Reviewed Data
• Session: E-Commerce and E-Content
• DETECTIVES: DETEcting Coalition hiT Inflation attacks in adVertising nEtworks Streams
• Extraction and Search of Chemical Formulae in Text Documents on the Web
• A Content-Driven Reputation System for the Wikipedia

Track: Industrial Practice & Experience

• Session: IPE
• Google News Personalization: Scalable Online Collaborative Filtering
• Exploring in the Weblog Space by Detecting Informative and Affective Articles
• Spam Double-Funnel: Connecting Web Spammers with Advertisers

Track: Performance and Scalability

• Session: Scalable Systems for Dynamic Content
• GlobeTP: Template-Based Database Replication for Scalable Web Applications
• Consistency-preserving Caching of Dynamic Database Content
• Optimized Query Planning of Continuous Aggregation Queries in Dynamic Data Dissemination Networks
• Session: Performance Engineering of Web Applications
• A Scalable Application Placement Controller for Enterprise Data Centers
• A Unified Platform for Data Driven Web Applications with Automatic Client-Server Partitioning
• Is High-Quality VoD Feasible using P2P Swarming?

Track: Pervasive Web and Mobility

• Session: Pervasive Web and Mobility
• Robust Web Page Segmentation for Mobile Terminal Using Content-Distances and Page Layout Information
• PRIVE: Anonymous Location-Based Queries in Distributed Mobile Systems
• A Mobile Application Framework for the Geospatial Web

Track: Search

• Session: Crawlers
• The Discoverability of the Web
• Combining Classifiers to Identify Online Databases
• An Adaptive Crawler for Locating Hidden-Web Entry Points
• Session: Web Graph
• Random Web Crawls
• Extraction and classification of dense communities in the Web
• Web Projections: Learning from Contextual Subgraphs of the Web
• Session: Search Quality and Precision
• Supervised Rank Aggregation
• Navigating the intranet with high precision
• Optimizing Web Search Using Social Annotation
• Session: Knowledge Discovery
• Compare&Contrast: Using the Web to Discover Comparable Cases for News Stories
• Answering Bounded Continuous Search Queries in the World Wide Web
• Answering Relationship Queries on the Web
• Session: Advertisements and Click Estimates
• Robust Methodologies for Modeling Web Click Distributions
• Predicting Clicks: Estimating the Click-Through Rate for New Ads
• Dynamics of bid optimization in online advertisement auctions
• Session: Search Potpourri
• Navigation-Aided Retrieval
• Efficient Search Engine Measurements
• Efficient Search in Large Textual Collections with Redundancy
• Session: Personalization
• Dynamic Personalized Pagerank in Entity-Relation Graphs
• A Large-scale Evaluation and Analysis of Personalized Search Strategies
• Privacy-Enhancing Personalized Web Search

Track: Security, Privacy, Reliability and Ethics

• Session: Passwords and Phishing
• CANTINA: A Content-Based Approach to Detecting Phishing Web Sites
• Learning to Detect Phishing Emails
• A Large-Scale Study of Web Password Habits
• Session: Defending Against Emerging (and Emerged) Threats
• Defeating Script Injection Attacks with Browser-Enforced Embedded Policies
• Subspace: Secure Cross-Domain Communication for Web Mashups
• Exposing Private Information by Timing Web Applications
• On Anonymizing Query Logs via Token-based Hashing
• Session: Access Control and Trust on the Web
• A Fault Model and Mutation Testing of Access Control Policies
• Analyzing Web Access Control Policies
• Compiling Cryptographic Protocols for Deployment on the Web

Track: Semantic Web

• Session: Ontologies
• Yago: A Core of Semantic Knowledge - Unifying WordNet and Wikipedia
• Ontology Summarization Based on RDF Sentence Graph
• Just the Right Amount: Extracting Modules from Ontologies
• Session: Similarity and Extraction
• Measuring Semantic Similarity between Words Using Web Search Engines
• Using Google Distance to weight approximate ontology matches
• Hierarchical Perceptron-like Learning for Ontology-Based Information Extraction
• Session: Query Languages and DBs
• From SPARQL to Rules (and back)
• SPARQ2L: Towards Support For Subgraph Extraction Queries in RDF Databases
• Bridging the Gap Between OWL and Relational Databases
• ActiveRDF: Object-Oriented Semantic Web Programming
• Session: Applications
• Towards Expressive Syndication on the Web
• Exhibit: Light-weight Structured Data Publishing
• Explorations in the use of Semantic Web Technologies for Product Information Management
• Session: Semantic Web and Web 2.0
• The two cultures -- position paper
• Analysis of Topological Characteristics of Huge Online Social Networking Services
• P-TAG: Large Scale Automatic Generation of Personalized Annotation TAGs for the Web

Track: Technology for Developing Regions

• Session: Communication in Developing Regions
• Connecting the bottom of the pyramid: an exploratory case study of India's rural communication environment
• Communication as Information-Seeking: The Case for Mobile Social Software for Developing Regions
• Optimal Audio-Visual Representations for Illiterate Users of Computers
• Session: Networking Issues in the Web
• Identifying and Discriminating Between Web and Peer-to-Peer Traffic in the Network Core
• Long Distance Wireless Mesh Network Planning: Problem Formulation and Solution
• MyXDNS: A Request Routing DNS Server With Decoupled Server Selection

Track: Web Engineering

• Session: Web Modeling
• Turning portlets into services: introducing the organization profile
• A Framework for Rapid Integration of Presentation Components
• Integrating Value-based Requirement Engineering Models to WebML using VIP Business Modeling Framework
• Session: End-Users Perspective and Measurement in Web Engineering
• Towards Effective Browsing of Large Scale Social Annotations
• Supporting End-Users in the Creation of Dependable Web Clips
• Effort Estimation: How Valuable is it for a Web company to Use a Cross-company Data Set Compared to Using Its Own Single-company Data Set?

Track: Web Services

• Session: Orchestration & Choreography
• Towards the Theoretical Foundation of Choreography
• Introduction and Evaluation of Martlet: A Scientific Workflow Language for Abstracted Parallelisation
• Semi-Automated Adaptation of Service Interactions
• Session: SLAs and QoS
• Reliable QoS Monitoring Based on Client Feedback
• Preference-based Selection of Highly Configurable Web Services
• Speeding up Adaptation of Web Service Compositions Using Expiration Times
• DIANE - An Integrated Approach to Automated Service Discovery Matchmaking and Composition

Track: XML and Web Data

• Session: Querying and Transforming XML
• Multiway SLCA-based Keyword Search in XML Data
• Visibly Pushdown Automata for Streaming XML
• Mapping-Driven XML Transformation
• Session: Parsing, Normalizing, and Storing XML
• Querying and Maintaining a Compact XML Storage
• XML Design for Relational Storage
• A High-Performance Interpretive Approach to Schema-Directed Parsing

FreeType + CImg 汉字渲染引擎

在一幅图像上显示一个汉字，看似很简单，其实如果不用所谓的WinAPI，还不是很简单。
首先我们需要处理字库文件，FreeType是一个开源的字库软件，可以读取Microsoft的TrueType矢量字库。如果要处理汉字，一般使用Unicode编码。

有了字库处理包，我们还需要一个强大的图像开发库。OpenCV是一个不错的选择，但是这个库从安装到使用都不是很方便。为此我选用了CImg的库，这个库如此简单，以至于它只有一个头文件。其他什么都没有。而且CImg是完全基于C++的，使用了template技术。而且它支持几乎所有的文件格式，这是很多做图像的人所希望的。

下面这个字使用上面的两个库画出来的：

汉字编码 C++

我用C++基于STL写了一个汉字编码的处理库。
可以实现 编码转换，自动编码检测。
重载了wstring的函数，使用很方便。

需要的可以和我联系：xlvector@gmail.com

目前只支持UTF-8和GBK编码。

UTF-8编码

UTF-8编码和Unicode编码其实没有区别，只是为了方便机器识别和移植，换了一种表现方式。也就是说Unicode和GBK之间的转换不需要用转换表，而是写一个程序就行了。

关于UTF-8和Unicode的介绍，可以看这儿。

关于如何由Unicode编码为UTF-8的方法网上很多，下面转载一个：

UTF-8 有一下特性:

• UCS 字符 U+0000 到 U+007F (ASCII) 被编码为字节 0x00 到 0x7F (ASCII 兼容). 这意味着只包含 7 位 ASCII 字符的文件在 ASCII 和 UTF-8 两种编码方式下是一样的.
• 所有 >U+007F 的 UCS 字符被编码为一个多个字节的串, 每个字节都有标记位集. 因此, ASCII 字节 (0x00-0x7F) 不可能作为任何其他字符的一部分.
• 表示非 ASCII 字符的多字节串的第一个字节总是在 0xC0 到 0xFD 的范围里, 并指出这个字符包含多少个字节. 多字节串的其余字节都在 0x80 到 0xBF 范围里. 这使得重新同步非常容易, 并使编码无国界, 且很少受丢失字节的影响.
• 可以编入所有可能的 2³¹个 UCS 代码
• UTF-8 编码字符理论上可以最多到 6 个字节长, 然而 16 位 BMP 字符最多只用到 3 字节长.
• Bigendian UCS-4 字节串的排列顺序是预定的.
• 字节 0xFE 和 0xFF 在 UTF-8 编码中从未用到.

下列字节串用来表示一个字符. 用到哪个串取决于该字符在 Unicode 中的序号.

U-00000000 - U-0000007F:	0xxxxxxx
U-00000080 - U-000007FF:	110xxxxx 10xxxxxx
U-00000800 - U-0000FFFF:	1110xxxx 10xxxxxx 10xxxxxx
U-00010000 - U-001FFFFF:	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
U-00200000 - U-03FFFFFF:	111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
U-04000000 - U-7FFFFFFF:	1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

xxx 的位置由字符编码数的二进制表示的位填入. 越靠右的 x 具有越少的特殊意义. 只用最短的那个足够表达一个字符编码数的多字节串. 注意在多字节串中, 第一个字节的开头"1"的数目就是整个串中字节的数目.

例如: Unicode 字符 U+00A9 = 1010 1001 (版权符号) 在 UTF-8 里的编码为:

11000010 10101001 = 0xC2 0xA9

而字符 U+2260 = 0010 0010 0110 0000 (不等于) 编码为:

11100010 10001001 10100000 = 0xE2 0x89 0xA0

这种编码的官方名字拼写为 UTF-8, 其中 UTF 代表 UCS Transformation Format. 请勿在任何文档中用其他名字 (比如 utf8 或 UTF_8) 来表示 UTF-8, 当然除非你指的是一个变量名而不是这种编码本身.

汉字编码

最近在写一个通用的类解决不同汉字编码的字符串的问题。这个问题困扰我们已久。这次终于下决心彻底解决这个问题。
sigh,为什么要有那么多种编码。全世界都统一起来多好，嘿嘿。