Convert to fpp
This commit is contained in:
Greg Turnquist
parent
1a9af53840
commit
64d031939f
|
|
@ -0,0 +1,119 @@
|
|||
<#assign project_id="gs-batch-processing">
|
||||
|
||||
# Getting Started: Creating a Batch Service
|
||||
|
||||
What you'll build
|
||||
-----------------
|
||||
|
||||
This guide walks you through creating a basic batch-driven solution. You build a service that imports data from a CSV spreadsheet, transforms it with custom code, and stores the final results in a database.
|
||||
|
||||
What you'll need
|
||||
----------------
|
||||
|
||||
- About 15 minutes
|
||||
- <@prereq_editor_jdk_buildtools/>
|
||||
|
||||
## <@how_to_complete_this_guide jump_ahead='Create a business class'/>
|
||||
|
||||
|
||||
<a name="scratch"></a>
|
||||
Set up the project
|
||||
------------------
|
||||
<@build_system_intro/>
|
||||
|
||||
<@create_directory_structure_hello/>
|
||||
|
||||
### Create a Maven POM
|
||||
|
||||
<@snippet path="pom.xml" prefix="complete"/>
|
||||
|
||||
<@bootstrap_starter_pom_disclaimer/>
|
||||
|
||||
### Create business data
|
||||
|
||||
Typically your customer or a business analyst supplies a spreadsheet. In this case, you make it up.
|
||||
|
||||
<@snippet path="src/main/resources/sample-data.csv" prefix="initial"/>
|
||||
|
||||
This spreadsheet contains a first name and a last name on each row, separated by a comma. This is a fairly common pattern that Spring handles out-of-the-box, as you will see.
|
||||
|
||||
### Define the destination for your data
|
||||
|
||||
Next, you write a SQL script to create a table to store the data.
|
||||
|
||||
<@snippet path="src/main/resources/schema.sql" prefix="initial"/>
|
||||
|
||||
<a name="initial"></a>
|
||||
Create a business class
|
||||
-----------------------
|
||||
|
||||
Now that you see the format of data inputs and outputs, you write code to represent a row of data.
|
||||
|
||||
<@snippet path="src/main/java/hello/Person.java" prefix="complete"/>
|
||||
|
||||
You can instantiate the `Person` class either with first and last name through a constructor, or by setting the properties.
|
||||
|
||||
Create an intermediate processor
|
||||
--------------------------------
|
||||
|
||||
A common paradigm in batch processing is to ingest data, transform it, and then pipe it out somewhere else. Here you write a simple transformer that converts the names to uppercase.
|
||||
|
||||
<@snippet path="src/main/java/hello/PersonItemProcessor.java" prefix="complete"/>
|
||||
|
||||
`PersonItemProcessor` implements Spring Batch's `ItemProcessor` interface. This makes it easy to wire the code into a batch job that you define further down in this guide. According to the interface, you receive an incoming `Person` object, after which you transform it to an upper-cased `Person`.
|
||||
|
||||
> **Note:** There is no requirement that the input and output types be the same. In fact, after one source of data is read, sometimes the application's data flow needs a different data type.
|
||||
|
||||
Put together a batch job
|
||||
----------------------------
|
||||
|
||||
Now you put together the actual batch job. Spring Batch provides many utility classes that reduce the need to write custom code. Instead, you can focus on the business logic.
|
||||
|
||||
<@snippet path="src/main/java/hello/BatchConfiguration.java" prefix="complete"/>
|
||||
|
||||
For starters, the `@EnableBatchProcessing` annotation adds many critical beans that support jobs and saves you a lot of leg work.
|
||||
|
||||
Break it down:
|
||||
|
||||
<@snippet "src/main/java/hello/BatchConfiguration.java" "readerwriterprocessor" "/complete"/>
|
||||
|
||||
The first chunk of code defines the input, processor, and output.
|
||||
- `reader()` creates an `ItemReader`. It looks for a file called `sample-data.csv` and parses each line item with enough information to turn it into a `Person`.
|
||||
- `processor()` creates an instance of our `PersonItemProcessor` you defined earlier, meant to uppercase the data.
|
||||
- `write(DataSource)` creates an `ItemWriter`. This one is aimed at a JDBC destination and automatically gets a copy of the dataSource created by `@EnableBatchProcessing`. It includes the SQL statement needed to insert a single `Person` driven by java bean properties.
|
||||
|
||||
The next chunk focuses on the actual job configuration.
|
||||
|
||||
<@snippet "src/main/java/hello/BatchConfiguration.java" "jobstep" "/complete"/>
|
||||
|
||||
The first method defines the job and the second one defines a single step. Jobs are built from steps, where each step can involve a reader, a processor, and a writer.
|
||||
|
||||
In this job definition, you need an incrementer because jobs use a database to maintain execution state. You then list each step, of which this job has only one step. The job ends, and the java API produces a perfectly configured job.
|
||||
|
||||
In the step definition, you define how much data to write at a time. In this case, it writes up to ten records at a time. Next, you configure the reader, processor, and writer using the injected bits from earlier.
|
||||
|
||||
> **Note:** chunk() is prefixed `<Person,Person>` because it's a generic method. This represents the input and output types of each "chunk" of processing, and lines up with `ItemReader<Person>` and `ItemWriter<Person>`.
|
||||
|
||||
Finally, you run the application.
|
||||
|
||||
<@snippet "src/main/java/hello/BatchConfiguration.java" "templatemain" "/complete"/>
|
||||
|
||||
This example uses a memory-based database (provided by `@EnableBatchProcessing`), meaning that when it's done, the data is gone. For demonstration purposes, there is extra code to create a `JdbcTemplate`, query the database, and print out the names of people the batch job inserts.
|
||||
|
||||
## <@build_an_executable_jar/>
|
||||
|
||||
|
||||
Run the batch job
|
||||
-----------------
|
||||
|
||||
Now you can run the job from the JAR as well, and distribute that as an executable artifact:
|
||||
|
||||
$ java -jar target/${project_id}-complete-0.1.0.jar
|
||||
|
||||
|
||||
The job prints out a line for each person that gets transformed. After the job runs, you can also see the output from querying the database.
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
Congratulations! You built a batch job that ingested data from a spreadsheet, processed it, and wrote it to a database.
|
||||
72
README.md
72
README.md
|
|
@ -1,3 +1,4 @@
|
|||
|
||||
# Getting Started: Creating a Batch Service
|
||||
|
||||
What you'll build
|
||||
|
|
@ -26,11 +27,13 @@ To **start from scratch**, move on to [Set up the project](#scratch).
|
|||
To **skip the basics**, do the following:
|
||||
|
||||
- [Download][zip] and unzip the source repository for this guide, or clone it using [git](/understanding/git):
|
||||
`git clone https://github.com/springframework-meta/{@project-name}.git`
|
||||
- cd into `{@project-name}/initial`
|
||||
- Jump ahead to [Create a resource representation class](#initial).
|
||||
`git clone https://github.com/springframework-meta/gs-batch-processing.git`
|
||||
- cd into `gs-batch-processing/initial`
|
||||
- Jump ahead to [Create a business class](#initial).
|
||||
|
||||
**When you're finished**, you can check your results against the code in `gs-batch-processing/complete`.
|
||||
[zip]: https://github.com/springframework-meta/gs-batch-processing/archive/master.zip
|
||||
|
||||
**When you're finished**, you can check your results against the code in `{@project-name}/complete`.
|
||||
|
||||
<a name="scratch"></a>
|
||||
Set up the project
|
||||
|
|
@ -56,7 +59,7 @@ In a project directory of your choosing, create the following subdirectory struc
|
|||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<groupId>org.springframework</groupId>
|
||||
<artifactId>gs-batch-processing-initial</artifactId>
|
||||
<artifactId>gs-batch-processing-complete</artifactId>
|
||||
<version>0.1.0</version>
|
||||
|
||||
<parent>
|
||||
|
|
@ -76,6 +79,19 @@ In a project directory of your choosing, create the following subdirectory struc
|
|||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<properties>
|
||||
<start-class>hello.BatchConfiguration</start-class>
|
||||
</properties>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<id>spring-snapshots</id>
|
||||
|
|
@ -100,15 +116,14 @@ In a project directory of your choosing, create the following subdirectory struc
|
|||
|
||||
TODO: mention that we're using Spring Bootstrap's [_starter POMs_](../gs-bootstrap-starter) here.
|
||||
|
||||
> Note to experienced Maven users who don't use an external parent project: You can take out the project later, it's just there to reduce the amount of code you have to write to get started.
|
||||
|
||||
Note to experienced Maven users who are unaccustomed to using an external parent project: you can take it out later, it's just there to reduce the amount of code you have to write to get started.
|
||||
|
||||
### Create business data
|
||||
|
||||
Typically your customer or a business analyst supplies a spreadsheet. In this case, you make it up.
|
||||
|
||||
`src/main/resources/sample-data.csv`
|
||||
```text
|
||||
```csv
|
||||
Jill,Doe
|
||||
Joe,Doe
|
||||
Justin,Doe
|
||||
|
|
@ -423,30 +438,37 @@ Finally, you run the application.
|
|||
|
||||
This example uses a memory-based database (provided by `@EnableBatchProcessing`), meaning that when it's done, the data is gone. For demonstration purposes, there is extra code to create a `JdbcTemplate`, query the database, and print out the names of people the batch job inserts.
|
||||
|
||||
### Build an executable JAR
|
||||
|
||||
Build an executable JAR
|
||||
-----------------------
|
||||
Add the following to your `pom.xml` file, keeping existing properties and plugins intact:
|
||||
Now that your `Application` class is ready, you simply instruct the build system to create a single, executable jar containing everything. This makes it easy to ship, version, and deploy the service as an application throughout the development lifecycle, across different environments, and so forth.
|
||||
|
||||
Add the following configuration to your existing Maven POM:
|
||||
|
||||
`pom.xml`
|
||||
```xml
|
||||
<properties>
|
||||
<start-class>hello.BatchConfiguration</start-class>
|
||||
</properties>
|
||||
<properties>
|
||||
<start-class>hello.Application</start-class>
|
||||
</properties>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
```
|
||||
|
||||
Create a single executable JAR file containing all necessary dependency classes:
|
||||
The `start-class` property tells Maven to create a `META-INF/MANIFEST.MF` file with a `Main-Class: hello.Application` entry. This entry enables you to run the jar with `java -jar`.
|
||||
|
||||
$ mvn package
|
||||
The [Maven Shade plugin][maven-shade-plugin] extracts classes from all jars on the classpath and builds a single "über-jar", which makes it more convenient to execute and transport your service.
|
||||
|
||||
Now run the following to produce a single executable JAR file containing all necessary dependency classes and resources:
|
||||
|
||||
mvn package
|
||||
|
||||
[maven-shade-plugin]: https://maven.apache.org/plugins/maven-shade-plugin
|
||||
|
||||
|
||||
Run the batch job
|
||||
|
|
@ -463,5 +485,3 @@ Summary
|
|||
-------
|
||||
|
||||
Congratulations! You built a batch job that ingested data from a spreadsheet, processed it, and wrote it to a database.
|
||||
|
||||
[zip]: https://github.com/springframework-meta/gs-batch-processing/archive/master.zip
|
||||
|
|
|
|||
|
|
@ -24,7 +24,6 @@
|
|||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<!-- {!begin#shade-config} -->
|
||||
<properties>
|
||||
<start-class>hello.BatchConfiguration</start-class>
|
||||
</properties>
|
||||
|
|
@ -37,7 +36,6 @@
|
|||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
<!-- {!end#shade-config} -->
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ import org.springframework.jdbc.core.RowMapper;
|
|||
@EnableAutoConfiguration
|
||||
public class BatchConfiguration {
|
||||
|
||||
// {!begin#reader-writer-processor}
|
||||
// {!begin readerwriterprocessor}
|
||||
@Bean
|
||||
public ItemReader<Person> reader() {
|
||||
FlatFileItemReader<Person> reader = new FlatFileItemReader<Person>();
|
||||
|
|
@ -64,9 +64,9 @@ public class BatchConfiguration {
|
|||
writer.setDataSource(dataSource);
|
||||
return writer;
|
||||
}
|
||||
// {!end#reader-writer-processor}
|
||||
// {!end readerwriterprocessor}
|
||||
|
||||
// {!begin#job-step}
|
||||
// {!begin jobstep}
|
||||
@Bean
|
||||
public Job importUserJob(JobBuilderFactory jobs, Step s1) {
|
||||
return jobs.get("importUserJob")
|
||||
|
|
@ -86,9 +86,9 @@ public class BatchConfiguration {
|
|||
.writer(writer)
|
||||
.build();
|
||||
}
|
||||
// {!end#job-step}
|
||||
// {!end jobstep}
|
||||
|
||||
// {!begin#template-main}
|
||||
// {!begin templatemain}
|
||||
@Bean
|
||||
public JdbcTemplate jdbcTemplate(DataSource dataSource) {
|
||||
return new JdbcTemplate(dataSource);
|
||||
|
|
@ -106,5 +106,5 @@ public class BatchConfiguration {
|
|||
System.out.println("Found <" + person + "> in the database.");
|
||||
}
|
||||
}
|
||||
// {!end#template-main}
|
||||
// {!end templatemain}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue