Table of contents
Foreword:
Everyone knows that mongodb is a non-relational database, that is to say, each table in the mongodb database exists independently, and there is no dependency between tables. When we have a very large amount of data, it is not satisfactory to filter through filtering conditions; therefore, when we design, we directly distinguish through DB, such as: chat records, build DB through roomId, etc. In mongodb, in addition to various CRUD statements, it also provides us with aggregation functions. This article mainly talks about the operation of aggregate in mongodb.
Application scenario:
Recently, due to the development of live broadcast projects, MongoDB is used to store user chat data, and the amount of data in the later stage will be very large. Handle the data structure well, and it’s no problem to process hundreds of millions of data every day. The editor used Mongodb to store router access record data before, and the data exceeded 100 million per day.
MongoDB is a document-type database, which is divided into databases (DBs) and collections (Collections), which are equivalent to SQL databases and data tables respectively. Due to this business reason, the data is all query operations, and there are no operations such as addition, deletion, and modification.
Command description:
The concept of pipeline:
Pipelines are generally used in Unix and Linux to pass the output of the current command as an argument to the next command.
MongoDB's aggregation pipeline passes the results of MongoDB documents to the next pipeline after one pipeline is processed. Pipeline operations are repeatable.
Expressions: Process input documents and output. Expressions are stateless and can only be used to compute documents in the current aggregation pipeline, not other documents.
Here we introduce several commonly used operations in the aggregation framework:
● $project: (select) Modify the structure of the input document. Can be used to rename, add or delete fields, and can be used to create calculations and nested documents.
● $match: (where) is used to filter data and only output documents that meet the conditions. $match uses MongoDB's standard query operations.
● $group: (group by) groups the documents in the collection, which can be used for statistical results.
the ● $match: (having).
● $sum: (sum) can be used for statistical results.
● $sum $sortByCount: (count) can be used for statistics.
● $lookup: (join) to establish a connection.
● $limit: (limit) is used to limit the number of documents returned by the MongoDB aggregation pipeline.
● $skip: (offset) Skips the specified number of documents in the aggregation pipeline and returns the remaining documents.
● $unwind: Split an array type field in the document into multiple pieces, each containing a value in the array.
● $sort: (order by, 1 ascending order, -1 descending order) sorts the input documents and outputs them.
● $geoNear: output ordered documents close to a geographic location.
Code example:
It should be noted that MongoDB has a pipeline concept, which is equivalent to layer-by-layer filtering. The order cannot be disordered, otherwise an error will be reported (A pipeline stage specification object must contain exactly obe field). This is actually the same as when we use sql statements to query, it is the same as select.....where...group by...order by...limit...The order cannot be messed up.
Party inquiries:
// match (equivalent to WHERE or HAVING )
BasicDBObject query= new BasicDBObject();
BasicDBObject[] array = { new BasicDBObject("time", new BasicDBObject("$gte", "2018-09-12")),new BasicDBObject("time", new BasicDBObject("$lte","2018-12-25"))};
query.append("$and", array);
BasicDBObject match = new BasicDBObject("$match", query);
// group (equivalent to GROUP BY)
BasicDBObject group = new BasicDBObject("$group", new BasicDBObject("_id", "$subject")
.append("count", new BasicDBObject("$sum", 1)));
// sort
BasicDBObject sort = new BasicDBObject("$sort", new BasicDBObject("count", -1));//1: positive sequence, -1 reverse sequence
// skip (skip how many previous pieces of data, used when paging)
//limt (as long as the first number of data, used when paging)
BasicDBObject limit = new BasicDBObject("$limit", pageSize);
BasicDBObject skip = new BasicDBObject("$skip", (pageNo - 1) * pageSize);
// The **order in the queryList collection cannot be disordered**, otherwise an error will be reported.
List<DBObject> queryList = new ArrayList<>();
queryList .add(match);
queryList .add(group);
queryList .add(sort);
queryList .add(skip);
queryList .add(limit);
// Variable dbName, dynamically pointing to DB
AggregateIterable<Document> iterable = mongoClient.mongoClient.getDatabase(dbName).getCollection(gatherName).aggregate(queryList );
The above code is the operation after aggregation. There are two fields (_id,count) in the queried collection, the value corresponding to _id is subject, and the value corresponding to count is the total number, so we can find out the result and perform For the paging operation, you only need to pass in the corresponding pageNo and pageSize.
Quantity query:
Since we designed the room as the dimension roomID as the DB (variable DBName), the query quantity can be used in the following ways:
// Get the total number of chats Long count = mongoTemplate.getCollection(DBName).countDocuments();
If this article is helpful to you, I am very happy to help you.
Of course, if you feel that there is something in the article that makes you feel unreasonable, or there is an easier way to implement it, or there is something that you can’t understand, I hope you can point it out in the comments after reading it, and I will read it as soon as possible reply you.