Skyline thinking

 

 

We have finished testing and deployed the latest version of Individual Code Review. The limitation on the number of users has also been removed. Welcome to use and join Individual Code Review.

Individual Code Review

https://www.icodereview.dev/

 

Many space agencies have accumulated large amounts of satellite data. The Land Process Distributed Active Archive Centre (LP DAAC) stores more than 500TB remote sensing data. However, scientists cannot analyse these data efficiently, so they have to spend hours or days waiting for analysing results or only analyse a small part of these data. SHAHED, a MapReduce-based system, was developed to analyse these data and provide real-time querying services.

SHAHED includes two main types of components, data interface components and user interface components. Data interface components consist of uncertainty handling module and indexing module, but I consider that the MapReduce module, one of subprojects of Hadoop which is a distributed computation project, should be classified in these components. User interface components are composed of querying module, mining module which is not included in this paper, and visualization module. SHAHED also provide a web interface where user can query data and can achieve results of heat maps.

When SHAHED starts running, the indexing module builds 3 stock quad trees based on 1200, 2400 and 4800 resolutions. Each tree includes all coordinate information of the world and is completely stored in main memory. And then, this module uses different machines to build daily spatial indexes and compute aggregate values. Finally, if these spatial indexes include a whole month information, they are combined to generate a quad tree for the month. If the monthly trees include a whole year, a quad tree for the year is built. These daily indexes, monthly indexes and year indexes compose a temporal index.

When daily data is downloaded from LP DAAC, uncertainty module is activated. These data probably miss values because of 4 reasons, such as out of the earth, water areas, mis-alignment and clouds. Uncertainty module utilises a traditional linear interpolation function to compute estimated values. After this step, this module notifies the indexing module to build a daily index based on these recovered data. The indexing module will build a monthly index and a yearly index if it is necessary.

Querying module provides two types of real-time query services without MapReduce. The first type is selection queries which return all matching values in the certain time ranges and specific areas. These queries use the temporal index to get a selection list and use the quad trees to get the spatial ranges. Finally, this system retrieves the values from disks based on temporal ranges and spatial ranges. The second one is aggregate queries. The first and second steps are same as selection queries. In the last step, the aggregate values are computed. As a result, all queries are completed in 1 second.

Visualization module provides 3 types of results, including static images, videos and multi-level images, of heat maps with MapReduce. Firstly, this module gets specified time ranges and spatial ranges with the temporal index and the quad tress. After this step, the map functions of MapReduce are executed on many machines to calculate a small part of the heat maps. Finally, each reducer of MapReduce generates an image. The final results are generated with these images. According to the experiments of authors, this system can build a heat map of the whole world in 200 seconds.

To summarise, SHAHED utilises MapReduce and temporal-spatial indexes to achieve better performance when it processes hundreds of terabyte data.

Labels: MapReduce

 

"Motor Schema-Based Mobile Robot Navigation" written by Atlanta and Georgia in 1989 describes how schemas were used to navigate a pilot with Autonomous Robot Architecture (AuRA) and what are the results of a simulation and experiments. Schemas are a methodology to describe mobile robot systems’ reactions when these systems detect the real world. Every schema is a pattern of action, so many schemas can be combined to cope with complicated situations.

The theory of schemas derives from brain science, psychology, neuroscientific and robotics. Firstly, a study of the frog created a model for worm acquisition, which was developed by Arbib and House in 1987. This model is based on visual orientation and suitable for mobile robot’s domain. In addition, the agent theory in artificial intelligence domain was combined with this model. As a result, a schema is designed as an agent with the preying model. A schema instantiation (SI) is created as an agent instance when an event occurs and parameters of this event are put into the schema. In particular, every schema is a distributed computing unit. Therefore, combined schemas are able to track the individual obstacles and to follow roads and leaders.

There are two types of schemas: motor schemas, such as stay-on–path schemas, avoid-static-obstacles schemas and avoid-moving-obstacles schemas, and perceptual schemas, like find-intersection schemas and find-landmark schemas. The most significant point is that every motor schema must be connected to a perceptual schema. Robots rely on these SIs to interact with their environment and these SIs must communicate with each other as these communication can avoid dead spots. On the other hand, to improve the computing performance, robots only compute the limited field in the current perception rather than the entire potential field.

The authors performed a simulation and many experiments in seven different situations, including avoidance, exploration, hall following, the presence of obstacles, single wall following, impatient waiting and following the leader situations. The simulation was run on a VAX 750 with a 64 units’ grid. As a result, the robot arrived at its goal and avoided all of obstacles. In these experiments, a robot named HARV, a Denning Research Vehicle, was developed with ultrasonic devices. HARV was able to seek an appropriate path using the avoid-static-obstacle schema and the noise schema in avoidance and exploration situations. In hall following, the presence of obstacles and single wall following situations, the move-ahead schema, the move-to-goal schema and avoid-static-obstacle schema navigated the robot to the goals. In last two situations, the robot followed the moving leader and went through the door when the door was opened. However, there were some issues in the presence of obstacles situation because HARV had to wait for processing results of sensing data.

To summarise, this schema-based navigation worked well in the simulation and the experiments. Schemas are able to be combined in different situations and this modular construction is easy to utilise and develop. In addition, this navigation system has better performance as the distributed computation agents utilise resources efficiently.

Your personal data will be processed and information from your device (cookies, unique identifiers, and other device data) may be stored by, accessed by and shared with 0 TCF vendor(s), or used specifically by this site or app.

Some vendors may process your personal data on the basis of legitimate interest, which you can object to by managing your options below. Look for a link at the bottom of this page to manage or withdraw consent in privacy and cookie settings.

You can choose how your personal data is used.

Vendors can use your data to provide services. Declining a vendor can stop them from using the data you shared.

We won't sell or share your personal information to inform the ads you see. You may still see interest-based ads if your information is sold or shared by other companies or was sold or shared previously.

 

“The Scrum software development process for small teams” written by Linda Rising and Norman S. Janoff in 2000 describes experiences using Scrum methodology in three teams. The authors introduced this methodology into their development teams of AG Communication Systems and these projects were successful.

The Scrum is an iterative and incremental development methodology and an iteration is called a sprint which has a fixed time box from one week to four weeks. This methodology is suitable for small teams in which the number of developers is no more than 10.

A Scrum team includes three types of members: a marketing group, a development group and a Scrum master. The marketing group represent clients, so they decide the features of software and have rights to modify requirements and cancel a sprint or a project. The development group implement the features defined by marketing group, but these two groups must work together as they need to discuss priorities of features and estimate the workload in a sprint. Therefore, all team members agree with the defined goal and they focus on the goal in the sprint. The Scrum master is a servant-leader who helps all members understand the Scrum process and regulations, and outside groups or organizations understand what the team are doing.

The Scrum process consists of an initial phase and some sprints. In initial phase, the product backlog which defines most features of projects and a basic architecture are produced, but this backlog and architecture are able to be modified in following sprints. In each sprint, there are many activities, including planning meeting, daily meetings, development and reviews. The Planning meeting provides a sprint backlog which is extracted from the product backlog and all members focus on this backlog. All members share their progress about the sprint backlog and obstacles in daily meetings. All information in a project is transparent, so every member understands requirements and progress of the sprint. After development of a sprint, stakeholders, customers and team members attend a review meeting where this project will be estimated and planed.

The authors provide feedbacks of three teams. There are some successful experiences and issues needed to be tackled. A team developed a switch simulator and this simulator was delivered on time. The main gains are how to define an achievable backlog and realise the importance of daily meetings. However, they did not control the time of daily meeting, which needs to be improved. B team and C team developed a new product and new feature respectively. They got feedbacks rapidly and benefited from these short iterations. As a result, three teams managed to complete their projects and wanted to utilize the Scrum in next projects.

To summarize, the Scrum methodology divides projects into many sprints and emphasizes communication among team members. These strategies help projects achieve their goals and are efficient.

Labels: Scrum