Archive

Posts Tagged ‘IQ’

Sybase IQ Installation

March 24th, 2012 No comments

Hi All,

Please find attached Sybase IQ Installation on Unix/Linux platform.

Sybase_IQ_installation

Petabyte Size Data Store Managed by Hadoop & Map Reduce.

February 11th, 2012 No comments

Hadoop
——–

Source : http://hadoop.apache.org/ & www.
Today, we’re surrounded by data. People upload videos, take pictures on their cell phones, text friends, update their Facebook status, leave comments around the web, click on ads, and so forth. Machines, too, are generating and keeping more and more data. You may even be reading this book as digital data on your computer screen, and certainly your purchase of this book is recorded as data with some retailer.

The exponential growth of data first presented challenges to cutting-edge businesses such as Google, Yahoo, Amazon, and Microsoft. They needed to go through terabytes and petabytes of data to figure out which websites were popular, what books were in demand, and what kinds of ads appealed to people. Existing tools were becoming inadequate to process such large data sets. Google was the first to publicize MapReduce—a system they had used to scale their data processing needs.

This system aroused a lot of interest because many other businesses were facing similar scaling challenges, and it wasn’t feasible for everyone to reinvent their own proprietary tool. Doug Cutting saw an opportunity and led the charge to develop an open source version of this MapReduce system called Hadoop . Soon after, Yahoo and others rallied around to support this effort.

What is Hadoop ?
————–

Hadoop is an open source framework for writing and running distributed applications that process large amounts of data. Distributed computing is a wide and varied field, but the key distinctions of Hadoop are that it is

1.Accessible—Hadoop runs on large clusters of commodity machines or on cloud computing services such as Amazon’s Elastic Compute Cloud (EC2 ).
2.Robust—Because it is intended to run on commodity hardware, Hadoop is architected with the assumption of frequent hardware malfunctions. It can gracefully handle most such failures.
3.Scalable—Hadoop scales linearly to handle larger data by adding more nodes to the cluster.
4.Simple—Hadoop allows users to quickly write efficient parallel code.

Comparing SQL databases and Hadoop:
————————————

Hadoop is a framework for processing data, what makes it better than standard relational databases, the workhorse of data processing in most of today’s applications? One reason is that SQL (structured query language) is by design targeted at structured data. Many of Hadoop’s initial applications deal with unstructured data such as text. From this perspective Hadoop provides a more general paradigm than SQL.
For working only with structured data, the comparison is more nuanced. In principle, SQL and Hadoop can be complementary, as SQL is a query language which can be implemented on top of Hadoop as the execution engine.3 But in practice, SQL databases tend to refer to a whole set of legacy technologies, with several dominant vendors, optimized for a historical set of applications. Many of these existing commercial databases are a mismatch to the requirements that Hadoop targets.
Some Implementation of Hadoop for production purpose :
——————————————————

Complete List @ http://wiki.apache.org/hadoop/PoweredBy

Sybase IQ
———
Sybase IQ : http://www.computerworld.com/s/article/9221355/Updated_Sybase_IQ_supports_Hadoop_MapReduce_Big_Data_

EBay
—-

532 nodes cluster (8 * 532 cores, 5.3PB).
Heavy usage of Java MapReduce, Pig, Hive, HBase
Using it for Search optimization and Research.

Facebook
——-

We use Hadoop to store copies of internal log and dimension data sources and use it as a source for reporting/analytics and machine learning.

Currently we have 2 major clusters:

A 1100-machine cluster with 8800 cores and about 12 PB raw storage.
A 300-machine cluster with 2400 cores and about 3 PB raw storage.
Each (commodity) node has 8 cores and 12 TB of storage.
We are heavy users of both streaming as well as the Java APIs. We have built a higher level data warehousing framework using these features called Hive (see the http://hadoop.apache.org/hive/). We have also developed a FUSE implementation over HDFS.

LinkedIn
———

We have multiple grids divided up based upon purpose. * Hardware:
120 Nehalem-based Sun x4275, with 2×4 cores, 24GB RAM, 8x1TB SATA
580 Westmere-based HP SL 170x, with 2×4 cores, 24GB RAM, 6x2TB SATA
1200 Westmere-based SuperMicro X8DTT-H, with 2×6 cores, 24GB RAM, 6x2TB SATA
Software:
CentOS 5.5 -> RHEL 6.1
Sun JDK 1.6.0_14 -> Sun JDK 1.6.0_20 -> Sun JDK 1.6.0_26
Apache Hadoop 0.20.2+patches -> Apache Hadoop 0.20.204+patches
Pig 0.9 heavily customized
Azkaban for scheduling
Hive, Avro, Kafka, and other bits and pieces…

Twitter
——–

We use Hadoop to store and process tweets, log files, and many other types of data generated across Twitter. We use Cloudera’s CDH2 distribution of Hadoop, and store all data as compressed LZO files.

We use both Scala and Java to access Hadoop’s MapReduce APIs
We use Pig heavily for both scheduled and ad-hoc jobs, due to its ability to accomplish a lot with few statements.
We employ committers on Pig, Avro, Hive, and Cassandra, and contribute much of our internal Hadoop work to opensource (see hadoop-lzo)
For more on our use of Hadoop, see the following presentations: Hadoop and Pig at Twitter and Protocol Buffers and Hadoop at Twitter

Yahoo!
——–

More than 100,000 CPUs in >40,000 computers running Hadoop
Our biggest cluster: 4500 nodes (2*4cpu boxes w 4*1TB disk & 16GB RAM)
Used to support research for Ad Systems and Web Search
Also used to do scaling tests to support development of Hadoop on larger clusters
Our Blog – Learn more about how we use Hadoop.
>60% of Hadoop Jobs within Yahoo are Pig jobs.

 

Data_

Categories: News, Sybase IQ Server Tags: , ,

Sybase IQ 15 Best Practices Guide

December 4th, 2011 No comments

What’s in YOUR Architecture?

December 4th, 2011 No comments

What’s in your architecture?

Probably not enough if you’re using just data modeling to integrate all the components of your enterprise, especially your information. To lay the
foundation of this paper, let’s start from the same level of understanding: the need for a common approach to managing all the aspects of information to
enable rapid business performance in the 21st century economy.

Competition in the marketplace is always fierce. To stay abreast, organizations must constantly analyze their customer needs and expectations, enhance
or innovate their business processes, and deliver products and services that create exceptional customer value. Organizations also need to be customercentric to forge long-term relationships with clients and consumers. Only organizations that are agile enough to respond to volatile market conditions
with innovation, expedited time-to-market processes, and reduced costs can differentiate themselves from the competition. Such agility occurs when a
company’s IT operations are closely aligned with its business operations. IT needs to understand business to implement technologies and applications that
support the current and future business goals.

Read Full Article @ http://www.sybase.com/files/White_Papers/Sybase_Whats_in_Your_Architecture_WP.pdf

bcp copy in failed

August 17th, 2011 No comments

Hi Folks,

Few days back , we  have faced the issue regarding bcp in,  message was bcp copy in failed. Although Rows were inserting successfully.  The user which We were using that was dbo of database and had all
permissions.

We tried with other login id and it was working fine, that was dbo alias.

We tried to remove user, added as  dbo alias, given explicit permission,but it didn’t work out.

It was totally weried as there was no problem at all , or you can say the issue which we were not able to
investigate as error is not much explanatory.

Also I would like to mention , we have refreshed this database from 12.5.4 to 15.0.3 env.

Finally we dropped the user, dropped login account and added again , means we craeted new login account
with same name  and after that added in db as a dbo alias. It worked fine now.
I am still not sure why this caused the issue.

Categories: ASE Tags: , , , ,

If failures is my Destiny, I would like to top in failures!!

August 7th, 2011 No comments

–If 100 plans fail, will make 200 next day!

— If failures is my Destiny, I would like to top in failures!!

Categories: ASE Tags: , , , ,

Sybase: PowerBuilder and PowerDesigner

July 27th, 2011 No comments
  • Sybase PowerBuilder is a RAD tool that lets you develop rich GUI applications,

based on the DataWindow concept.I’ve been told that part of the

SAP POS module is actually developed  with PowerBuilder.

  • Sybase PowerDesigner is a modeling tool that lets you handle anything from a 

data model to a full enterprise architecture, and is widely seen as

one of the best such tools around.

Source : http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/25286
Both PowerBuilder and PowerDesigner are not tied to any specific database and will work with most common database brands.

 Read more...
Categories: ASE Tags: , , , ,

17700 + Hits in sybaseblog.com!

July 26th, 2011 No comments

Hi Guys,

Today your blog crossed the  17700+ hits!!

I would like to congratulate and thanks for your continued support !!!

Thanks once again!

Cheers!!

AnVa

Categories: ASE Tags: , , , ,

Happy Holi to All of you!

March 19th, 2011 No comments

Happy Holi to All of you!

Categories: ASE Tags: , , , ,

Load database: A Recurring Issue!

February 24th, 2011 5 comments

Hi Guys,

As we already aware for loading a database , there cann’t be any user online in the DB.

In our env, we are killing these spids before actual database load in script.

Sometime it works, sometime fails as clinet ids logged in so frequently in server so during the actual load statement database again comes in use status.

This is very frequent issue which I have seen in many enviorment where clined ids logged in very rapidally.

For that we using the some alternative as locking the ids manually and once the load starts we unlock.( We cannt wait untill load finish for unlock of ids) . This task required the manual intervention.

I would like to raise a question, why sybase cann’t provide the load database with kill option, beacuse any above such alternative would not be 100% effective as killing the spids ( by any way) and loading the database is two diffrent task. Between both task execution, there must be some time gap may be in millsecond, and that time gap is suffcient for any login id to logged ins.

One more option can be :

We can make database in offline state( something like that so no one can use it) before load database, aftre that we will go for load database.

Kindly let me know your thoughts and suggest if any thing I am missing above.

I have posted the same in various forums as well, will update you if I’ll get any response.

till then..

bye

Categories: ASE Tags: , , , ,