Two possibilities...

[Randy Carson]

That's the thing, they didn't. They used the authority of Jesus supposedly passed on to peter and every other 'pope' from Him to the one who decided to wage war against the muslims or jews or whom ever. The problem with that? The word in which Jesus used to identify 'peter' is not one of authority. Jesus was actually being a little insulting to peter/petros (translated: loose gravel/unsure footing/small stone) when He told Him of the 'rock'/petra in which He would build His church.

Opponents of the Catholic interpretation of Matthew 16:18 sometimes argue that in the Greek text the name of the apostle is Petros, while "rock" is rendered as petra. They claim that the former refers to a small stone, while the latter refers to a massive rock; so, if Peter was meant to be the massive rock, why isn’t his name Petra?

Note that Christ did not speak to the disciples in Greek. He spoke Aramaic, the common language of Palestine at that time. In that language the word for rock is kepha, which is what Jesus called him in everyday speech (note that in John 1:42 he was told, "You will be called Cephas"). What Jesus said in Matthew 16:18 was: "You are Kepha, and upon this kepha I will build my Church."

When Matthew’s Gospel was translated from the original Aramaic to Greek, there arose a problem which did not confront the evangelist when he first composed his account of Christ’s life. In Aramaic the word kepha has the same ending whether it refers to a rock or is used as a man’s name. In Greek, though, the word for rock, petra, is feminine in gender. The translator could use it for the second appearance of kepha in the sentence, but not for the first because it would be inappropriate to give a man a feminine name. So he put a masculine ending on it, and hence Peter became Petros.

Furthermore, the premise of the argument against Peter being the rock is simply false. In first century Greek the words petros and petra were synonyms. They had previously possessed the meanings of "small stone" and "large rock" in some early Greek poetry, but by the first century this distinction was gone, as Protestant Bible scholars admit (see D. A. Carson’s remarks on this passage in the Expositor’s Bible Commentary, [Grand Rapids: Zondervan Books]).

Some of the effect of Christ’s play on words was lost when his statement was translated from the Aramaic into Greek, but that was the best that could be done in Greek. In English, like Aramaic, there is no problem with endings; so an English rendition could read: "You are Rock, and upon this rock I will build my church."

Consider another point: If the rock really did refer to Christ (as some claim, based on 1 Cor. 10:4, "and the Rock was Christ" though the rock there was a literal, physical rock), why did Matthew leave the passage as it was? In the original Aramaic, and in the English which is a closer parallel to it than is the Greek, the passage is clear enough. Matthew must have realized that his readers would conclude the obvious from "Rock . . . rock."

If he meant Christ to be understood as the rock, why didn’t he say so? Why did he take a chance and leave it up to Paul to write a clarifying text? This presumes, of course, that 1 Corinthians was written after Matthew’s Gospel; if it came first, it could not have been written to clarify it.

The reason, of course, is that Matthew knew full well that what the sentence seemed to say was just what it really was saying. It was Simon, weak as he was, who was chosen to become the rock and thus the first link in the chain of the papacy.

Besides all of that, Peter did not start the R/C Church, Peter did not have a big gentile ministry, and nothing ever written places him in Rome.

and

Peter did not start the Church. Jesus did, and He promised to build His Church (singular, not plural) upon Peter, the rock. Dozens of Protestant and Orthodox scholars have acknowledged that Peter IS the rock.

But yes, Peter was in Rome.

1 Peter 5:13:
She who is at Babylon, who is likewise chosen, sends you greetings; and so does my son Mark.

“Babylon” was an early Christian reference for “Rome,” so Sts. Peter and Mark are sending their greetings from Rome.

Second, this is also the testimony of the Church Fathers, who testify that Mark is Peter's disciple and interpreter in Rome. St. Irenaeus, writing c. 180 A.D., says:

Matthew also issued a written Gospel among the Hebrews in their own dialect, while Peter and Paul were preaching at Rome, and laying the foundations of the Church. After their departure, Mark, the disciple and interpreter of Peter, did also hand down to us in writing what had been preached by Peter.

Eusebius says the same thing, as does St. Jerome:

Mark, the disciple and interpreter of Peter wrote a short gospel at the request of the brethren at Rome embodying what he had heard Peter tell. When Peter had heard this, he approved it and published it to the churches to be read by his authority as Clemens in the sixth book of his Hypotyposes and Papias, bishop of Hierapolis, record. Peter also mentions this Mark in his first epistle, figuratively indicating Rome under the name of Babylon "She who is in Babylon elect together with you salutes you and so does Mark my son." So, taking the gospel which he himself composed, he went to Egypt and first preaching Christ at Alexandria he formed a church so admirable in doctrine and continence of living that he constrained all followers of Christ to his example.

It was Paul who started the church at rome,

If you read the Letter to the Romans, you will discover that Paul is writing to a well-established Church...a Church which he has never visited...though he longs to do so on his way to Spain (cf. Rm 15:24). This is why Paul mentions so many people by name...he is establishing his own credentials as an apostle to a church probably founded by believers who were in Jerusalem on the day of Pentecost and were led by Apollos or possibly Aquila and Priscilla (cf. Rom 16) and saying, "Look, I know many of the same people you do."

and nothing else ever written by Peter or Paul hinted at Apostolic powers/Authority being passed down.

Many people deny that the modern Catholic Church is the one Church Jesus promised to build (cf. Mt. 16:18-19) claiming that the doctrine of Apostolic Succession is not found in the Bible. Is this argument valid?

Let’s begin by examining the evidence contained in scripture as well as the non-scriptural writings of the earliest Christians for evidence of Apostolic Succession. The Bible contains clear indications that the Apostle Paul taught Apostolic Succession to his disciples and fellow workers, Timothy, Titus and Clement. Here are the relevant passages:

2 Timothy 2:1-2
You then, my son, be strong in the grace that is in Christ Jesus. And the things you have heard me say in the presence of many witnesses entrust to reliable men who will also be qualified to teach others.

There are four generations of believers contained in this one passage: 1. Paul himself, 2. Timothy, who was Paul’s disciple, 3. Those whom Timothy would disciple, and 4. Those to whom Timothy’s disciples would preach. Paul commanded Timothy to hand on the gospel to reliable men and further to ensure that those men would also hand on the gospel reliably.

Titus 1:5
The reason I left you in Crete was that you might straighten out what was left unfinished and appoint elders in every town, as I directed you.

In the passage above, we see that Paul was concerned with the appointing of capable leaders in the Cretan church. So in addition to his concern for the content of the message, he is concerned with the succession of the leadership, as well.

Paul also outlined the beginnings of Church hierarchy (as well as the qualification for Church office) in his first letter to Timothy.

“Here is a trustworthy saying: If anyone sets his heart on being an overseer, he desires a noble task. Now the overseer (bishop) must be above reproach, the husband of but one wife, temperate, self-controlled, respectable, hospitable, able to teach, not given to drunkenness, not violent but gentle, not quarrelsome, not a lover of money. He must manage his own family well and see that his children obey him with proper respect. (If anyone does not know how to manage his own family, how can he take care of God's church?) He must not be a recent convert, or he may become conceited and fall under the same judgment as the devil. He must also have a good reputation with outsiders, so that he will not fall into disgrace and into the devil's trap.

“Deacons, likewise, are to be men worthy of respect, sincere, not indulging in much wine, and not pursuing dishonest gain. They must keep hold of the deep truths of the faith with a clear conscience. They must first be tested; and then if there is nothing against them, let them serve as deacons.” (1 Timothy 3:1-10)

These verses illustrate that by the time this letter was written in the late first century, the Church had already established several positions of leadership: Apostles (Peter being the foremost among them), Overseers (or Bishops) and Deacons.

Philippians 4:3
Yes, and I ask you, loyal yokefellow, help these women who have contended at my side in the cause of the gospel, along with Clement and the rest of my fellow workers, whose names are in the book of life.

In the passage from Philippians, Paul mentions one of his fellow workers, Clement, who was ordained by the Apostle Peter and later became the fourth Bishop of Rome (after Peter, Linus, and Anacletus). Like Paul, who addressed to epistles to the Church of Corinth, Clement wrote his own letter to the Corinthians around 80 AD. In that letter, he stated:

"Through countryside and city [the apostles] preached, and they appointed their earliest converts, testing them by the Spirit, to be the bishops and deacons of future believers. Nor was this a novelty, for bishops and deacons had been written about a long time earlier. . . . Our apostles knew through our Lord Jesus Christ that there would be strife for the office of bishop. For this reason, therefore, having received perfect foreknowledge, they appointed those who have already been mentioned and afterwards added the further provision that, if they should die, other approved men should succeed to their ministry" (Letter to the Corinthians 42:4–5, 44:1–3 [A.D. 80]).

“We are of opinion, therefore, that those appointed by them, or afterwards by other eminent men, with the consent of the whole church, and who have blamelessly served the flock of Christ, in a humble, peaceable, and disinterested spirit, and have for a long time possessed the good opinion of all, cannot be justly dismissed from the ministry. (ibid.)

From these two passages, we can see that Clement had witnessed his mentors, the Apostles Peter and Paul, naming men to the office of Bishop and had received instructions from them that other men should succeed those Bishops appointed by the Apostles in the event that these first Bishops should die. Thus, history records that both the Apostles and their disciples such as Clement, Timothy and Titus understood and followed the practice of appointing successors to the Apostles in the Church.

While many seem to believe that anyone with a Bible may become a “pastor” by simply gathering around himself a group of fellow believers to form a church, the Bible itself teaches that true leaders in the Church of Jesus Christ must be ordained by those who were ordained before them. This process, known as Apostolic Succession, maintains an unbroken chain of continuity from Jesus, Peter and the Apostles to the leaders of the early Church.